[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:12 dist-test-slave-2w3w krb5kdc[28780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:12 dist-test-slave-2w3w krb5kdc[28780](info): set up 2 sockets
Aug 11 11:15:12 dist-test-slave-2w3w krb5kdc[28780](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260811 11:15:13.140776 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:13 dist-test-slave-2w3w krb5kdc[28780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:15:13.283121 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:43079
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:36211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:43079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:13.376566 28793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:13.376807 28793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:13.376833 28793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:13.378153 28793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:13.378207 28793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:13.378221 28793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:13.378234 28793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:13.378247 28793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:13.380105 28793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:43079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:43079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:13.380507 28793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:13.380810 28793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:15:13.388480 28793 server_base.cc:1034] running on GCE node
W20260811 11:15:13.389721 28798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:13.389743 28799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:13.390127 28801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:13.390542 28793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:13.391108 28793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:13.392285 28793 hybrid_clock.cc:648] HybridClock initialized: now 1786446913392258 us; error 79 us; skew 500 ppm
Aug 11 11:15:13 dist-test-slave-2w3w krb5kdc[28780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446913, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:13.400346 28793 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:15:13.401203 28793 webserver.cc:460] Webserver started at http://127.28.25.62:40415/ using document root <none> and password file <none>
I20260811 11:15:13.401594 28793 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:13.401687 28793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:13.401922 28793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:13.403501 28793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "42d59a22ee49462bb55946cc127b47e0"
format_stamp: "Formatted at 2026-08-11 11:15:13 on dist-test-slave-2w3w"
server_key: "8770eb4d5fbb0edfebb4c749c022dfb1"
server_key_iv: "be14352a72223d802a49dea715996861"
server_key_version: "encryptionkey@0"
I20260811 11:15:13.404246 28793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "42d59a22ee49462bb55946cc127b47e0"
format_stamp: "Formatted at 2026-08-11 11:15:13 on dist-test-slave-2w3w"
server_key: "8770eb4d5fbb0edfebb4c749c022dfb1"
server_key_iv: "be14352a72223d802a49dea715996861"
server_key_version: "encryptionkey@0"
I20260811 11:15:13.406657 28793 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:15:13.408244 28810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:13.408587 28793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:13.408736 28793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "42d59a22ee49462bb55946cc127b47e0"
format_stamp: "Formatted at 2026-08-11 11:15:13 on dist-test-slave-2w3w"
server_key: "8770eb4d5fbb0edfebb4c749c022dfb1"
server_key_iv: "be14352a72223d802a49dea715996861"
server_key_version: "encryptionkey@0"
I20260811 11:15:13.408874 28793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:13.434810 28793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:13.437654 28793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:13.437949 28793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:13.443513 28793 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:43079
I20260811 11:15:13.443604 28867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:43079 every 8 connection(s)
I20260811 11:15:13.444252 28793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:13.445667 28869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:13.449196 28869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0: Bootstrap starting.
I20260811 11:15:13.450316 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 28793
I20260811 11:15:13.450376 28869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:13.450460 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:13.450856 28869 log.cc:824] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:13.451339 28772 external_mini_cluster.cc:1428] Setting key ad5ac167759124f5c19eed63ea08f59b
I20260811 11:15:13.451915 28869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0: No bootstrap required, opened a new log
I20260811 11:15:13.454587 28869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d59a22ee49462bb55946cc127b47e0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43079 } }
I20260811 11:15:13.454885 28869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:13.455047 28869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d59a22ee49462bb55946cc127b47e0, State: Initialized, Role: FOLLOWER
I20260811 11:15:13.455262 28869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d59a22ee49462bb55946cc127b47e0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43079 } }
I20260811 11:15:13.455421 28869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:13.455489 28869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:13.455543 28869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:13.456926 28869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d59a22ee49462bb55946cc127b47e0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43079 } }
I20260811 11:15:13.457119 28869 leader_election.cc:302] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42d59a22ee49462bb55946cc127b47e0; no voters:
I20260811 11:15:13.457355 28869 leader_election.cc:288] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:13.457433 28875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:13.457615 28875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [term 1 LEADER]: Becoming Leader. State: Replica: 42d59a22ee49462bb55946cc127b47e0, State: Running, Role: LEADER
I20260811 11:15:13.457764 28875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d59a22ee49462bb55946cc127b47e0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43079 } }
I20260811 11:15:13.457806 28869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:13.458339 28875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42d59a22ee49462bb55946cc127b47e0. Latest consensus state: current_term: 1 leader_uuid: "42d59a22ee49462bb55946cc127b47e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d59a22ee49462bb55946cc127b47e0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43079 } } }
I20260811 11:15:13.458442 28875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:13.459066 28876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42d59a22ee49462bb55946cc127b47e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d59a22ee49462bb55946cc127b47e0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43079 } } }
I20260811 11:15:13.459159 28876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:13.459419 28881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:13.460670 28881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:13.463640 28881 catalog_manager.cc:1347] Generated new cluster ID: 8f71b23ef20447329458668b97d43d90
I20260811 11:15:13.463732 28881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 11:15:13 dist-test-slave-2w3w krb5kdc[28780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:13.478212 28881 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:13.478904 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:13.452644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.28.25.62:43079)
0811 11:15:13.453025 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:13.453036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:13.456274 (+ 3238us) server_negotiation.cc:407] Connection header received
0811 11:15:13.459281 (+ 3007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:13.459329 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:13.459462 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:13.459660 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:13.463126 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:13.465423 (+ 2297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:13.467319 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:13.468201 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:13.468389 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:13.473545 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:13.473551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:13.473551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:13.473565 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:13.475992 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:13.476770 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:13.476773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:13.476774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:13.476927 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:13.477315 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:13.477316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:13.477316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:13.477612 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:13.477854 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:13.478158 (+ 304us) server_negotiation.cc:299] Negotiation successful
0811 11:15:13.478504 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
I20260811 11:15:13.479029 28881 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:13.484565 28881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42d59a22ee49462bb55946cc127b47e0: Generated new TSK 0
I20260811 11:15:13.484807 28881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:15:13.522454 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43079
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:36211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:13.627396 28899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:13.627643 28899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:13.627674 28899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:13.629249 28899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:13.629315 28899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:13.629351 28899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:13.631331 28899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:13.631678 28899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:13.631978 28899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:13.639837 28904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:13.639830 28905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:13.639983 28907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:13.640089 28899 server_base.cc:1034] running on GCE node
I20260811 11:15:13.640439 28899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:13.640736 28899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:13.641937 28899 hybrid_clock.cc:648] HybridClock initialized: now 1786446913641919 us; error 35 us; skew 500 ppm
Aug 11 11:15:13 dist-test-slave-2w3w krb5kdc[28780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446913, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:13.644893 28899 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:15:13.645331 28899 webserver.cc:460] Webserver started at http://127.28.25.1:36229/ using document root <none> and password file <none>
I20260811 11:15:13.645538 28899 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:13.645579 28899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:13.645675 28899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:13.646785 28899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "2ee1fd511bef4819abd338f5b289af94"
format_stamp: "Formatted at 2026-08-11 11:15:13 on dist-test-slave-2w3w"
server_key: "48e1f0316f5085226116a62f23ac9287"
server_key_iv: "f651cc974a487248011a39b7e73783e5"
server_key_version: "encryptionkey@0"
I20260811 11:15:13.647202 28899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "2ee1fd511bef4819abd338f5b289af94"
format_stamp: "Formatted at 2026-08-11 11:15:13 on dist-test-slave-2w3w"
server_key: "48e1f0316f5085226116a62f23ac9287"
server_key_iv: "f651cc974a487248011a39b7e73783e5"
server_key_version: "encryptionkey@0"
I20260811 11:15:13.648883 28899 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 11:15:13.649748 28916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:13.649983 28899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:13.650182 28899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "2ee1fd511bef4819abd338f5b289af94"
format_stamp: "Formatted at 2026-08-11 11:15:13 on dist-test-slave-2w3w"
server_key: "48e1f0316f5085226116a62f23ac9287"
server_key_iv: "f651cc974a487248011a39b7e73783e5"
server_key_version: "encryptionkey@0"
I20260811 11:15:13.650269 28899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:13.663229 28899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:13.665994 28899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:13.666302 28899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:13.666740 28899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:13.667315 28899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:13.667403 28899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:13.667433 28899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:13.667485 28899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:13.682066 28899 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:40215
I20260811 11:15:13.682193 29041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:40215 every 8 connection(s)
I20260811 11:15:13.682624 28899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:13.689865 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 28899
I20260811 11:15:13.689960 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:13.690110 28772 external_mini_cluster.cc:1428] Setting key 62cbda1b457aaf084b3c8c050986b8ad
Aug 11 11:15:13 dist-test-slave-2w3w krb5kdc[28780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446913, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:13.696349 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:13.683770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:54583 (local address 127.28.25.62:43079)
0811 11:15:13.683839 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:13.683840 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:13.684199 (+ 359us) server_negotiation.cc:407] Connection header received
0811 11:15:13.685113 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:13.685115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:13.685133 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:13.685197 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:13.686888 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:13.687474 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:13.688840 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:13.689084 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:13.689173 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:13.692566 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:13.692567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:13.692568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:13.692600 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:13.694345 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:13.694811 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:13.694812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:13.694813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:13.694894 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:13.695455 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:13.695456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:13.695456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:13.695647 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:13.695733 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:13.696111 (+ 378us) server_negotiation.cc:299] Negotiation successful
0811 11:15:13.696229 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260811 11:15:13.696535 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:13.683723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:43079 (local address 127.28.25.1:54583)
0811 11:15:13.684047 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:13.684088 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:13.684973 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:13.685270 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:13.685275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:13.686267 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:13.686632 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:13.686638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:13.687585 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:13.687586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:13.688722 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:13.688725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:13.689188 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:13.689189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:13.689421 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:13.690465 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:13.690475 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:13.692455 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:13.694451 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:13.694461 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:13.694463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:13.694724 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:13.695209 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:13.695210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:13.695210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:13.695372 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:13.695748 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:13.695757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:13.695936 (+ 179us) client_negotiation.cc:769] Sending connection context
0811 11:15:13.696063 (+ 127us) client_negotiation.cc:240] Negotiation successful
0811 11:15:13.696243 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":193,"threads_started":1}
I20260811 11:15:13.697386 29042 heartbeater.cc:347] Connected to a master server at 127.28.25.62:43079
I20260811 11:15:13.697492 29042 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:13.697718 29042 heartbeater.cc:511] Master 127.28.25.62:43079 requested a full tablet report, sending...
I20260811 11:15:13.699024 28830 ts_manager.cc:194] Registered new tserver with Master: 2ee1fd511bef4819abd338f5b289af94 (127.28.25.1:40215)
I20260811 11:15:13.700378 28830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:54583
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:15:13.727173 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43079
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:36211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:13.863308 29050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:13.863556 29050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:13.863588 29050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:13.864923 29050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:13.864969 29050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:13.865000 29050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:13.866763 29050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:13.867067 29050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:13.867301 29050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:13.874796 29055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:13.874831 29058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:13.874802 29056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:13.875003 29050 server_base.cc:1034] running on GCE node
I20260811 11:15:13.875288 29050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:13.875547 29050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:13.876735 29050 hybrid_clock.cc:648] HybridClock initialized: now 1786446913876727 us; error 42 us; skew 500 ppm
Aug 11 11:15:13 dist-test-slave-2w3w krb5kdc[28780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446913, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:13.880398 29050 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:15:13.880968 29050 webserver.cc:460] Webserver started at http://127.28.25.2:38995/ using document root <none> and password file <none>
I20260811 11:15:13.881207 29050 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:13.881264 29050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:13.881435 29050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:13.882792 29050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "d1de01c427c84ee3a4adfab9fd60b186"
format_stamp: "Formatted at 2026-08-11 11:15:13 on dist-test-slave-2w3w"
server_key: "d25240595b8f4ac8d2a2bab3c0913714"
server_key_iv: "2bddbc84f1e149b829caa744586a196a"
server_key_version: "encryptionkey@0"
I20260811 11:15:13.883306 29050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "d1de01c427c84ee3a4adfab9fd60b186"
format_stamp: "Formatted at 2026-08-11 11:15:13 on dist-test-slave-2w3w"
server_key: "d25240595b8f4ac8d2a2bab3c0913714"
server_key_iv: "2bddbc84f1e149b829caa744586a196a"
server_key_version: "encryptionkey@0"
I20260811 11:15:13.885123 29050 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260811 11:15:13.886166 29066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:13.886413 29050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 11:15:13.886536 29050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "d1de01c427c84ee3a4adfab9fd60b186"
format_stamp: "Formatted at 2026-08-11 11:15:13 on dist-test-slave-2w3w"
server_key: "d25240595b8f4ac8d2a2bab3c0913714"
server_key_iv: "2bddbc84f1e149b829caa744586a196a"
server_key_version: "encryptionkey@0"
I20260811 11:15:13.886624 29050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:13.899441 29050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:13.901834 29050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:13.902117 29050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:13.902576 29050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:13.903074 29050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:13.903131 29050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:13.903173 29050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:13.903203 29050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:13.921177 29050 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:41851
I20260811 11:15:13.921348 29190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:41851 every 8 connection(s)
I20260811 11:15:13.921679 29050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:15:13.925176 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 29050
I20260811 11:15:13.925259 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:13.925423 28772 external_mini_cluster.cc:1428] Setting key f8786a7371a560e2f8889099eabb1d3e
Aug 11 11:15:13 dist-test-slave-2w3w krb5kdc[28780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446913, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:13.935048 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:13.922609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:48497 (local address 127.28.25.62:43079)
0811 11:15:13.922671 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:13.922673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:13.923205 (+ 532us) server_negotiation.cc:407] Connection header received
0811 11:15:13.924122 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:13.924123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:13.924137 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:13.924196 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:13.925593 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:13.926242 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:13.927564 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:13.927775 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:13.927864 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:13.931608 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:13.931610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:13.931611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:13.931619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:13.933245 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:13.933863 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:13.933865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:13.933865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:13.933937 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:13.934240 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:13.934241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:13.934242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:13.934383 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:13.934466 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:13.934787 (+ 321us) server_negotiation.cc:299] Negotiation successful
0811 11:15:13.934900 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 11:15:13.935343 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:13.922586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:43079 (local address 127.28.25.2:48497)
0811 11:15:13.923041 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:13.923100 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:13.923991 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:13.924282 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:13.924286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:13.925022 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:13.925478 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:13.925487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:13.926365 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:13.926366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:13.927458 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:13.927461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:13.927882 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:13.927883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:13.928086 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:13.929005 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:13.929018 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:13.931440 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:13.933375 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:13.933390 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:13.933393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:13.933762 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:13.934021 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:13.934022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:13.934022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:13.934164 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:13.934456 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:13.934466 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:13.934636 (+ 170us) client_negotiation.cc:769] Sending connection context
0811 11:15:13.934750 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 11:15:13.934985 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":224,"threads_started":1}
I20260811 11:15:13.936007 29191 heartbeater.cc:347] Connected to a master server at 127.28.25.62:43079
I20260811 11:15:13.936168 29191 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:13.936479 29191 heartbeater.cc:511] Master 127.28.25.62:43079 requested a full tablet report, sending...
I20260811 11:15:13.936995 28830 ts_manager.cc:194] Registered new tserver with Master: d1de01c427c84ee3a4adfab9fd60b186 (127.28.25.2:41851)
I20260811 11:15:13.937641 28830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:48497
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:15:13.963972 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43079
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:36211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:14.109591 29197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:14.109874 29197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:14.109906 29197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:14.111527 29197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:14.111585 29197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:14.111617 29197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:14.113750 29197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:14.114069 29197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:14.114353 29197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:14.121730 29204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:14.121801 29203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:14.121769 29206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:14.122251 29197 server_base.cc:1034] running on GCE node
I20260811 11:15:14.122509 29197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:14.122864 29197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:14.124120 29197 hybrid_clock.cc:648] HybridClock initialized: now 1786446914124104 us; error 33 us; skew 500 ppm
Aug 11 11:15:14 dist-test-slave-2w3w krb5kdc[28780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446914, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:14.128154 29197 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:15:14.128667 29197 webserver.cc:460] Webserver started at http://127.28.25.3:37081/ using document root <none> and password file <none>
I20260811 11:15:14.128894 29197 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:14.128942 29197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:14.129099 29197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:14.130460 29197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "47b70d78123642e5a7e4bc1350efe10b"
format_stamp: "Formatted at 2026-08-11 11:15:14 on dist-test-slave-2w3w"
server_key: "8d1572cc915ba822e8cd4450beaefe29"
server_key_iv: "b8274622875579e5ef653c9a83099348"
server_key_version: "encryptionkey@0"
I20260811 11:15:14.130928 29197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "47b70d78123642e5a7e4bc1350efe10b"
format_stamp: "Formatted at 2026-08-11 11:15:14 on dist-test-slave-2w3w"
server_key: "8d1572cc915ba822e8cd4450beaefe29"
server_key_iv: "b8274622875579e5ef653c9a83099348"
server_key_version: "encryptionkey@0"
I20260811 11:15:14.133055 29197 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260811 11:15:14.134233 29213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:14.134488 29197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 11:15:14.134599 29197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "47b70d78123642e5a7e4bc1350efe10b"
format_stamp: "Formatted at 2026-08-11 11:15:14 on dist-test-slave-2w3w"
server_key: "8d1572cc915ba822e8cd4450beaefe29"
server_key_iv: "b8274622875579e5ef653c9a83099348"
server_key_version: "encryptionkey@0"
I20260811 11:15:14.134672 29197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:14.155186 29197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:14.157876 29197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:14.158015 29197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:14.158303 29197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:14.158676 29197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:14.158718 29197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:14.158766 29197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:14.158797 29197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:14.175349 29197 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:37839
I20260811 11:15:14.175581 29342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:37839 every 8 connection(s)
I20260811 11:15:14.175885 29197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:15:14.182760 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 29197
I20260811 11:15:14.182870 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:15:14.183090 28772 external_mini_cluster.cc:1428] Setting key a73f58e6bb718208c2e76e7a9484d403
Aug 11 11:15:14 dist-test-slave-2w3w krb5kdc[28780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446914, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:14.190089 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:14.177013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:39651 (local address 127.28.25.62:43079)
0811 11:15:14.177121 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:14.177124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:14.177523 (+ 399us) server_negotiation.cc:407] Connection header received
0811 11:15:14.178470 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:14.178471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:14.178486 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:14.178553 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:14.180141 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:14.180665 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:14.182262 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:14.182481 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:14.182598 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:14.186377 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:14.186379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:14.186379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:14.186390 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:14.188186 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:14.188709 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:14.188711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:14.188711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:14.188789 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:14.189160 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:14.189161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:14.189161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:14.189308 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:14.189382 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:14.189874 (+ 492us) server_negotiation.cc:299] Negotiation successful
0811 11:15:14.190015 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:15:14.190335 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:14.176920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:43079 (local address 127.28.25.3:39651)
0811 11:15:14.177345 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:14.177387 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:14.178343 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:14.178681 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:14.178685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:14.179648 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:14.180011 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:14.180018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:14.180860 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:14.180861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:14.182138 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:14.182142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:14.182628 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:14.182629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:14.182937 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:14.183939 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:14.183974 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:14.186200 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:14.188288 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:14.188300 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:14.188302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:14.188609 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:14.188925 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:14.188926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:14.188926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:14.189065 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:14.189437 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:14.189448 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:14.189661 (+ 213us) client_negotiation.cc:769] Sending connection context
0811 11:15:14.189838 (+ 177us) client_negotiation.cc:240] Negotiation successful
0811 11:15:14.190020 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":259,"threads_started":1}
I20260811 11:15:14.191054 29343 heartbeater.cc:347] Connected to a master server at 127.28.25.62:43079
I20260811 11:15:14.191159 29343 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:14.191394 29343 heartbeater.cc:511] Master 127.28.25.62:43079 requested a full tablet report, sending...
I20260811 11:15:14.191870 28830 ts_manager.cc:194] Registered new tserver with Master: 47b70d78123642e5a7e4bc1350efe10b (127.28.25.3:37839)
I20260811 11:15:14.192281 28830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:39651
I20260811 11:15:14.195719 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 11:15:14 dist-test-slave-2w3w krb5kdc[28780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 11:15:14 dist-test-slave-2w3w krb5kdc[28780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
I20260811 11:15:14.228088 29351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:14.218256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.28.25.1:40215)
0811 11:15:14.218448 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:14.218453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:14.218587 (+ 134us) server_negotiation.cc:407] Connection header received
0811 11:15:14.218640 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:14.218642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:14.218658 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:14.218765 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:14.219513 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:14.220530 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:14.221371 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:14.221818 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:14.221888 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:14.224599 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:14.224611 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:14.224611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:14.224619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:14.226671 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:14.227180 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:14.227183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:14.227184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:14.227248 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:14.227461 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:14.227462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:14.227462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:14.227625 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:14.227838 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:14.227889 (+ 51us) server_negotiation.cc:299] Negotiation successful
0811 11:15:14.227966 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
W20260811 11:15:14.228637 28948 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33334
Aug 11 11:15:14 dist-test-slave-2w3w krb5kdc[28780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 11:15:14 dist-test-slave-2w3w krb5kdc[28780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
I20260811 11:15:14.257638 29351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:14.248945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.28.25.1:40215)
0811 11:15:14.249051 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:14.249053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:14.249172 (+ 119us) server_negotiation.cc:407] Connection header received
0811 11:15:14.249209 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:14.249210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:14.249227 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:14.249292 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:14.250031 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:14.250550 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:14.251491 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:14.251726 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:14.251808 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:14.254382 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:14.254383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:14.254384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:14.254393 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:14.256114 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:14.256643 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:14.256644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:14.256645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:14.256719 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:14.256936 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:14.256937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:14.256937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:14.257055 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:14.257178 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:14.257305 (+ 127us) server_negotiation.cc:299] Negotiation successful
0811 11:15:14.257402 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 11:15:14.258376 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 28899
I20260811 11:15:14.267549 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 29050
I20260811 11:15:14.278487 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 29197
I20260811 11:15:14.286194 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 28793
2026-08-11T11:15:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2289 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:14 dist-test-slave-2w3w krb5kdc[29360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:14 dist-test-slave-2w3w krb5kdc[29360](info): set up 2 sockets
Aug 11 11:15:14 dist-test-slave-2w3w krb5kdc[29360](info): commencing operation
krb5kdc: starting...
W20260811 11:15:15.416257 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:15 dist-test-slave-2w3w krb5kdc[29360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:15:15.563158 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39857
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:39857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:15.654554 29375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:15.654827 29375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:15.654855 29375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:15.656183 29375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:15.656250 29375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:15.656265 29375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:15.656277 29375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:15.656289 29375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:15.658062 29375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:39857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:15.658390 29375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:15.658650 29375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:15.664012 29381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:15.664059 29380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:15.664012 29383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:15.664333 29375 server_base.cc:1034] running on GCE node
I20260811 11:15:15.664634 29375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:15.665017 29375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:15.666216 29375 hybrid_clock.cc:648] HybridClock initialized: now 1786446915666190 us; error 66 us; skew 500 ppm
Aug 11 11:15:15 dist-test-slave-2w3w krb5kdc[29360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446915, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:15.669579 29375 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:15:15.670239 29375 webserver.cc:460] Webserver started at http://127.28.25.62:40363/ using document root <none> and password file <none>
I20260811 11:15:15.670521 29375 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:15.670595 29375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:15.670761 29375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:15.672019 29375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "26157339226f4b849dc71c550942acd2"
format_stamp: "Formatted at 2026-08-11 11:15:15 on dist-test-slave-2w3w"
server_key: "4d04f72b26b5745fdd86c611a9152161"
server_key_iv: "33ced27bf703ddb446311b5a87835db9"
server_key_version: "encryptionkey@0"
I20260811 11:15:15.672534 29375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "26157339226f4b849dc71c550942acd2"
format_stamp: "Formatted at 2026-08-11 11:15:15 on dist-test-slave-2w3w"
server_key: "4d04f72b26b5745fdd86c611a9152161"
server_key_iv: "33ced27bf703ddb446311b5a87835db9"
server_key_version: "encryptionkey@0"
I20260811 11:15:15.674134 29375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:15:15.675118 29391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:15.675436 29375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:15.675619 29375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "26157339226f4b849dc71c550942acd2"
format_stamp: "Formatted at 2026-08-11 11:15:15 on dist-test-slave-2w3w"
server_key: "4d04f72b26b5745fdd86c611a9152161"
server_key_iv: "33ced27bf703ddb446311b5a87835db9"
server_key_version: "encryptionkey@0"
I20260811 11:15:15.675737 29375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:15.690006 29375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:15.692688 29375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:15.692893 29375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:15.697873 29375 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:39857
I20260811 11:15:15.697968 29446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:39857 every 8 connection(s)
I20260811 11:15:15.698434 29375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:15.699434 29447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:15.700001 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 29375
I20260811 11:15:15.700093 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:15.700268 28772 external_mini_cluster.cc:1428] Setting key 672edd010c9f5e75f7acec3b833f0b4b
I20260811 11:15:15.703507 29447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2: Bootstrap starting.
I20260811 11:15:15.704276 29447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:15.704618 29447 log.cc:824] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:15.705564 29447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2: No bootstrap required, opened a new log
Aug 11 11:15:15 dist-test-slave-2w3w krb5kdc[29360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:15.708221 29447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26157339226f4b849dc71c550942acd2" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39857 } }
I20260811 11:15:15.708428 29447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:15.708498 29447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26157339226f4b849dc71c550942acd2, State: Initialized, Role: FOLLOWER
I20260811 11:15:15.708684 29447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26157339226f4b849dc71c550942acd2" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39857 } }
I20260811 11:15:15.708788 29447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:15.708846 29447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:15.708915 29447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:15.709623 29447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26157339226f4b849dc71c550942acd2" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39857 } }
I20260811 11:15:15.709760 29447 leader_election.cc:302] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26157339226f4b849dc71c550942acd2; no voters:
I20260811 11:15:15.710028 29447 leader_election.cc:288] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:15.710081 29456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:15.710264 29456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [term 1 LEADER]: Becoming Leader. State: Replica: 26157339226f4b849dc71c550942acd2, State: Running, Role: LEADER
I20260811 11:15:15.710383 29447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:15.710443 29456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26157339226f4b849dc71c550942acd2" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39857 } }
I20260811 11:15:15.711032 29459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26157339226f4b849dc71c550942acd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26157339226f4b849dc71c550942acd2" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39857 } } }
I20260811 11:15:15.711201 29459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:15.710893 29456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26157339226f4b849dc71c550942acd2. Latest consensus state: current_term: 1 leader_uuid: "26157339226f4b849dc71c550942acd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26157339226f4b849dc71c550942acd2" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39857 } } }
I20260811 11:15:15.711544 29456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:15.711592 29467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:15.712235 29467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:15.712844 29451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:15.700871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.28.25.62:39857)
0811 11:15:15.701108 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:15.701117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:15.701148 (+ 31us) server_negotiation.cc:407] Connection header received
0811 11:15:15.701628 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:15.701636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:15.701692 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:15.701862 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:15.702575 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:15.703610 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:15.704451 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:15.704959 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:15.705050 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:15.707818 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:15.707825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:15.707826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:15.707846 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:15.710357 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:15.710861 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:15.710865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:15.710865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:15.711023 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:15.711254 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:15.711254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:15.711255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:15.711599 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:15.712305 (+ 706us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:15.712392 (+ 87us) server_negotiation.cc:299] Negotiation successful
0811 11:15:15.712551 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260811 11:15:15.714129 29467 catalog_manager.cc:1347] Generated new cluster ID: ecb37c30a1474d718529fc1d7694ed03
I20260811 11:15:15.714190 29467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:15.730706 29467 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:15.731446 29467 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:15.736737 29467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26157339226f4b849dc71c550942acd2: Generated new TSK 0
I20260811 11:15:15.737025 29467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:15:15.771469 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39857
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:15.863864 29480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:15.864121 29480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:15.864151 29480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:15.865489 29480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:15.865545 29480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:15.865577 29480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:15.867443 29480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:15.867692 29480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:15.867904 29480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:15.873502 29487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:15.873495 29486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:15.873495 29489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:15.874471 29480 server_base.cc:1034] running on GCE node
I20260811 11:15:15.874687 29480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:15.875032 29480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:15.876222 29480 hybrid_clock.cc:648] HybridClock initialized: now 1786446915876198 us; error 58 us; skew 500 ppm
Aug 11 11:15:15 dist-test-slave-2w3w krb5kdc[29360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446915, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:15.879485 29480 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:15:15.880132 29480 webserver.cc:460] Webserver started at http://127.28.25.1:35679/ using document root <none> and password file <none>
I20260811 11:15:15.880406 29480 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:15.880477 29480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:15.880638 29480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:15.881793 29480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "af8911a74ed247bcb87f31cf091d3178"
format_stamp: "Formatted at 2026-08-11 11:15:15 on dist-test-slave-2w3w"
server_key: "fe1ca9388aa3fc52bb726ce5df6362a4"
server_key_iv: "cb28973c0db8e58f30673c542acb0f47"
server_key_version: "encryptionkey@0"
I20260811 11:15:15.882445 29480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "af8911a74ed247bcb87f31cf091d3178"
format_stamp: "Formatted at 2026-08-11 11:15:15 on dist-test-slave-2w3w"
server_key: "fe1ca9388aa3fc52bb726ce5df6362a4"
server_key_iv: "cb28973c0db8e58f30673c542acb0f47"
server_key_version: "encryptionkey@0"
I20260811 11:15:15.884202 29480 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 11:15:15.885186 29497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:15.885419 29480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:15.885540 29480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "af8911a74ed247bcb87f31cf091d3178"
format_stamp: "Formatted at 2026-08-11 11:15:15 on dist-test-slave-2w3w"
server_key: "fe1ca9388aa3fc52bb726ce5df6362a4"
server_key_iv: "cb28973c0db8e58f30673c542acb0f47"
server_key_version: "encryptionkey@0"
I20260811 11:15:15.885629 29480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:15.916456 29480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:15.918814 29480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:15.919034 29480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:15.919312 29480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:15.919644 29480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:15.919697 29480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:15.919734 29480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:15.919778 29480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:15.926743 29480 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:39757
I20260811 11:15:15.926795 29622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:39757 every 8 connection(s)
I20260811 11:15:15.927227 29480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:15.928676 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 29480
I20260811 11:15:15.928789 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:15.929035 28772 external_mini_cluster.cc:1428] Setting key d4368312a089d678915846cff549488e
Aug 11 11:15:15 dist-test-slave-2w3w krb5kdc[29360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446915, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
I20260811 11:15:15.938323 29451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:15.928138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:59779 (local address 127.28.25.62:39857)
0811 11:15:15.928237 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:15.928239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:15.928439 (+ 200us) server_negotiation.cc:407] Connection header received
0811 11:15:15.929672 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:15.929673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:15.929686 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:15.929745 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:15.930466 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:15.931105 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:15.932199 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:15.932427 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:15.932527 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:15.935025 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:15.935027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:15.935027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:15.935035 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:15.936702 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:15.937216 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:15.937218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:15.937218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:15.937289 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:15.937586 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:15.937588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:15.937588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:15.937726 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:15.937812 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:15.938119 (+ 307us) server_negotiation.cc:299] Negotiation successful
0811 11:15:15.938222 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 11:15:15.938558 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:15.928081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39857 (local address 127.28.25.1:59779)
0811 11:15:15.928302 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:15.928343 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:15.929545 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:15.929841 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:15.929845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:15.930141 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:15.930360 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:15.930364 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:15.931212 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:15.931213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:15.932088 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:15.932090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:15.932545 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:15.932546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:15.932683 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:15.933313 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:15.933323 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:15.934862 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:15.936823 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:15.936835 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:15.936838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:15.937123 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:15.937371 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:15.937372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:15.937372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:15.937509 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:15.937800 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:15.937810 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:15.937980 (+ 170us) client_negotiation.cc:769] Sending connection context
0811 11:15:15.938098 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 11:15:15.938289 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":118,"threads_started":1}
I20260811 11:15:15.939448 29623 heartbeater.cc:347] Connected to a master server at 127.28.25.62:39857
I20260811 11:15:15.939637 29623 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:15.939879 29623 heartbeater.cc:511] Master 127.28.25.62:39857 requested a full tablet report, sending...
I20260811 11:15:15.940544 29408 ts_manager.cc:194] Registered new tserver with Master: af8911a74ed247bcb87f31cf091d3178 (127.28.25.1:39757)
I20260811 11:15:15.941314 29408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:59779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:15:15.963218 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39857
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:16.056524 29628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:16.056867 29628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:16.056919 29628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:16.059213 29628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:16.059289 29628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:16.059347 29628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:16.062575 29628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:16.062940 29628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:16.063299 29628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:16.069566 29635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:16.069574 29633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:16.069841 29639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:16.070008 29628 server_base.cc:1034] running on GCE node
I20260811 11:15:16.070289 29628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:16.070627 29628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:16.071821 29628 hybrid_clock.cc:648] HybridClock initialized: now 1786446916071788 us; error 46 us; skew 500 ppm
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446916, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:16.075704 29628 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:15:16.076494 29628 webserver.cc:460] Webserver started at http://127.28.25.2:39705/ using document root <none> and password file <none>
I20260811 11:15:16.076830 29628 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:16.076922 29628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:16.077099 29628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:16.078938 29628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "3dd7ebf1e3174204aa6c02944f64bb2b"
format_stamp: "Formatted at 2026-08-11 11:15:16 on dist-test-slave-2w3w"
server_key: "fd632025d058e6988a36c93ca5a897d3"
server_key_iv: "0be40988f1d8cfcc1113c3f5d7c7f6cb"
server_key_version: "encryptionkey@0"
I20260811 11:15:16.079491 29628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "3dd7ebf1e3174204aa6c02944f64bb2b"
format_stamp: "Formatted at 2026-08-11 11:15:16 on dist-test-slave-2w3w"
server_key: "fd632025d058e6988a36c93ca5a897d3"
server_key_iv: "0be40988f1d8cfcc1113c3f5d7c7f6cb"
server_key_version: "encryptionkey@0"
I20260811 11:15:16.081566 29628 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260811 11:15:16.082576 29646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:16.082805 29628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:16.082943 29628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "3dd7ebf1e3174204aa6c02944f64bb2b"
format_stamp: "Formatted at 2026-08-11 11:15:16 on dist-test-slave-2w3w"
server_key: "fd632025d058e6988a36c93ca5a897d3"
server_key_iv: "0be40988f1d8cfcc1113c3f5d7c7f6cb"
server_key_version: "encryptionkey@0"
I20260811 11:15:16.083069 29628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:16.099239 29628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:16.102001 29628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:16.102181 29628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:16.102475 29628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:16.102840 29628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:16.102878 29628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:16.103065 29628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:16.103103 29628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:16.109903 29628 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:36241
I20260811 11:15:16.109915 29774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:36241 every 8 connection(s)
I20260811 11:15:16.110373 29628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446916, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:16.120592 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 29628
I20260811 11:15:16.120726 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:16.120921 28772 external_mini_cluster.cc:1428] Setting key d7490a0ffa72ccb2a01ce3168f82bdf9
I20260811 11:15:16.122543 29451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.111285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:55103 (local address 127.28.25.62:39857)
0811 11:15:16.111356 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.111358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.111629 (+ 271us) server_negotiation.cc:407] Connection header received
0811 11:15:16.112880 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.112881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.112897 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.112955 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:16.113662 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.114178 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.115645 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.115878 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.115986 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.119077 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:16.119079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:16.119080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:16.119089 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:16.120547 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.121163 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.121165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.121165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.121392 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.121749 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.121751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.121751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.121914 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:16.121974 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.122379 (+ 405us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.122474 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:15:16.122884 29777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.111297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39857 (local address 127.28.25.2:55103)
0811 11:15:16.111495 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:16.111537 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:16.112743 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:16.113033 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:16.113038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:16.113334 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:16.113556 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.113560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.114298 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.114299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.115536 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.115539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.116010 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.116011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.116218 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.117047 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:16.117056 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:16.118869 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:16.120668 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:16.120680 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:16.120683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:16.121062 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:16.121494 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:16.121496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:16.121496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:16.121664 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:16.121970 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:16.121980 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:16.122216 (+ 236us) client_negotiation.cc:769] Sending connection context
0811 11:15:16.122348 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 11:15:16.122571 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260811 11:15:16.123539 29775 heartbeater.cc:347] Connected to a master server at 127.28.25.62:39857
I20260811 11:15:16.123664 29775 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:16.123893 29775 heartbeater.cc:511] Master 127.28.25.62:39857 requested a full tablet report, sending...
I20260811 11:15:16.124359 29408 ts_manager.cc:194] Registered new tserver with Master: 3dd7ebf1e3174204aa6c02944f64bb2b (127.28.25.2:36241)
I20260811 11:15:16.125102 29408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:55103
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:15:16.155283 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39857
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:16.245039 29780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:16.245275 29780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:16.245304 29780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:16.246575 29780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:16.246628 29780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:16.246658 29780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:16.248379 29780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:16.248668 29780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:16.248950 29780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:16.254673 29785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:16.254762 29780 server_base.cc:1034] running on GCE node
W20260811 11:15:16.254657 29786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:16.254657 29788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:16.255345 29780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:16.255659 29780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:16.256891 29780 hybrid_clock.cc:648] HybridClock initialized: now 1786446916256874 us; error 50 us; skew 500 ppm
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446916, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:16.260070 29780 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:15:16.260697 29780 webserver.cc:460] Webserver started at http://127.28.25.3:44249/ using document root <none> and password file <none>
I20260811 11:15:16.261010 29780 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:16.261092 29780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:16.261265 29780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:16.262571 29780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "67b6f9eddd854e4f8e0a88fc740f770d"
format_stamp: "Formatted at 2026-08-11 11:15:16 on dist-test-slave-2w3w"
server_key: "53921969e6cfa3af3a93cc1fce8706bd"
server_key_iv: "83d4cc16cee0b0dd732ee9ac74be9580"
server_key_version: "encryptionkey@0"
I20260811 11:15:16.263090 29780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "67b6f9eddd854e4f8e0a88fc740f770d"
format_stamp: "Formatted at 2026-08-11 11:15:16 on dist-test-slave-2w3w"
server_key: "53921969e6cfa3af3a93cc1fce8706bd"
server_key_iv: "83d4cc16cee0b0dd732ee9ac74be9580"
server_key_version: "encryptionkey@0"
I20260811 11:15:16.264796 29780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:16.265954 29795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:16.266202 29780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:16.266369 29780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "67b6f9eddd854e4f8e0a88fc740f770d"
format_stamp: "Formatted at 2026-08-11 11:15:16 on dist-test-slave-2w3w"
server_key: "53921969e6cfa3af3a93cc1fce8706bd"
server_key_iv: "83d4cc16cee0b0dd732ee9ac74be9580"
server_key_version: "encryptionkey@0"
I20260811 11:15:16.266489 29780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:16.312587 29780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:16.314903 29780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:16.315069 29780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:16.315302 29780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:16.315682 29780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:16.315721 29780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:16.315747 29780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:16.315809 29780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:16.323036 29780 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:43223
I20260811 11:15:16.323056 29921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:43223 every 8 connection(s)
I20260811 11:15:16.323508 29780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446916, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:16.332664 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 29780
I20260811 11:15:16.332799 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:15:16.333055 28772 external_mini_cluster.cc:1428] Setting key 79b83343cce5898510b9e635e4ad2c97
I20260811 11:15:16.334164 29451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.324299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:47181 (local address 127.28.25.62:39857)
0811 11:15:16.324352 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.324354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.324598 (+ 244us) server_negotiation.cc:407] Connection header received
0811 11:15:16.325652 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.325653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.325665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.325725 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:16.326422 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.326940 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.328112 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.328310 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.328367 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.330737 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:16.330739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:16.330739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:16.330752 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:16.332236 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.332870 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.332872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.332872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.332955 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.333285 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.333286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.333287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.333468 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:16.333541 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.333957 (+ 416us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.334064 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:15:16.334429 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.324286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39857 (local address 127.28.25.3:47181)
0811 11:15:16.324460 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:16.324504 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:16.325536 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:16.325789 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:16.325793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:16.326102 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:16.326338 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.326344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.327072 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.327073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.328021 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.328024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.328400 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.328401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.328533 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.329054 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:16.329063 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:16.330628 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:16.332340 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:16.332349 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:16.332351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:16.332755 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:16.333033 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:16.333034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:16.333035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:16.333213 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:16.333549 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:16.333557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:16.333785 (+ 228us) client_negotiation.cc:769] Sending connection context
0811 11:15:16.333934 (+ 149us) client_negotiation.cc:240] Negotiation successful
0811 11:15:16.334143 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260811 11:15:16.335058 29922 heartbeater.cc:347] Connected to a master server at 127.28.25.62:39857
I20260811 11:15:16.335180 29922 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:16.335418 29922 heartbeater.cc:511] Master 127.28.25.62:39857 requested a full tablet report, sending...
I20260811 11:15:16.335856 29408 ts_manager.cc:194] Registered new tserver with Master: 67b6f9eddd854e4f8e0a88fc740f770d (127.28.25.3:43223)
I20260811 11:15:16.336206 29408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:47181
I20260811 11:15:16.345268 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:15:16.353905 29451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.347418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.28.25.62:39857)
0811 11:15:16.347500 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.347503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.347601 (+ 98us) server_negotiation.cc:407] Connection header received
0811 11:15:16.347700 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.347701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.347712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.347779 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:16.348402 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.348934 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.349689 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.349903 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.349991 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.351249 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:16.351251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:16.351251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:16.351259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:16.352586 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.352986 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.352988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.352988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.353053 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.353267 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.353268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.353268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.353441 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:16.353553 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.353695 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.353786 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 11:15:16.357154 29408 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:15:16.358937 29408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:16.367934 29934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.362360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:39757 (local address 127.0.0.1:52598)
0811 11:15:16.362841 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:16.362851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:16.362922 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:16.363248 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:16.363252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:16.363255 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:16.363354 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.363356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.364970 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.364971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.366413 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.366417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.367559 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.367560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.367655 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.367671 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 11:15:16.367754 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 11:15:16.367824 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":37,"thread_start_us":104,"threads_started":1}
I20260811 11:15:16.368299 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.362660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.28.25.1:39757)
0811 11:15:16.362937 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.362940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.362949 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:16.363056 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.363058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.363092 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.363226 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:16.363445 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.364727 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.366532 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.367461 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.367543 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.367608 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.368047 (+ 439us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.368110 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
I20260811 11:15:16.368798 29933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.362345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:36241 (local address 127.0.0.1:34934)
0811 11:15:16.362564 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:16.362629 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:16.362702 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:16.363278 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:16.363279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:16.363281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:16.363397 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.363400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.364896 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.364897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.366424 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.366425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.368432 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.368433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.368588 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.368593 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:15:16.368655 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 11:15:16.368709 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":62,"thread_start_us":73,"threads_started":1}
I20260811 11:15:16.368945 29935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.362416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:43223 (local address 127.0.0.1:46828)
0811 11:15:16.363029 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:16.363038 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:16.363080 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:16.365589 (+ 2509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:16.365590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:16.365593 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:16.365786 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.365788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.366810 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.366811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.367675 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.367677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.368657 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.368658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.368737 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.368742 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:15:16.368791 (+ 49us) client_negotiation.cc:240] Negotiation successful
0811 11:15:16.368834 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":70,"threads_started":1}
I20260811 11:15:16.369087 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.362510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.28.25.2:36241)
0811 11:15:16.362904 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.362910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.362921 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:15:16.362994 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.362996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.363014 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.363488 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:16.363632 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.364730 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.367055 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.367994 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.368433 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.368513 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.368923 (+ 410us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.368975 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":135,"threads_started":1}
I20260811 11:15:16.369257 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.362645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.28.25.3:43223)
0811 11:15:16.365133 (+ 2488us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.365138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.365147 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:16.365248 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.365251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.365270 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.365353 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:16.365876 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.366677 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.367782 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.368481 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.368554 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.368614 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.369109 (+ 495us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.369163 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2423,"thread_start_us":111,"threads_started":1}
I20260811 11:15:16.369218 29549 tablet_service.cc:1467] Processing CreateTablet for tablet 1e9b424eaf0a46318e46bd38300ca683 (DEFAULT_TABLE table=test-table [id=739aeacbbe1349568bafe9c54277cb8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:16.369726 29549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e9b424eaf0a46318e46bd38300ca683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:16.370060 29701 tablet_service.cc:1467] Processing CreateTablet for tablet 1e9b424eaf0a46318e46bd38300ca683 (DEFAULT_TABLE table=test-table [id=739aeacbbe1349568bafe9c54277cb8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:16.370113 29851 tablet_service.cc:1467] Processing CreateTablet for tablet 1e9b424eaf0a46318e46bd38300ca683 (DEFAULT_TABLE table=test-table [id=739aeacbbe1349568bafe9c54277cb8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:16.370420 29701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e9b424eaf0a46318e46bd38300ca683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:16.370488 29851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e9b424eaf0a46318e46bd38300ca683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:16.374867 29939 tablet_bootstrap.cc:492] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178: Bootstrap starting.
I20260811 11:15:16.375274 29940 tablet_bootstrap.cc:492] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d: Bootstrap starting.
I20260811 11:15:16.376144 29941 tablet_bootstrap.cc:492] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b: Bootstrap starting.
I20260811 11:15:16.376561 29940 tablet_bootstrap.cc:654] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:16.376618 29939 tablet_bootstrap.cc:654] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:16.377887 29940 log.cc:824] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:16.378270 29939 log.cc:824] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:16.378302 29941 tablet_bootstrap.cc:654] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:16.379391 29940 tablet_bootstrap.cc:492] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d: No bootstrap required, opened a new log
I20260811 11:15:16.379542 29940 ts_tablet_manager.cc:1397] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260811 11:15:16.379812 29941 log.cc:824] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:16.379880 29939 tablet_bootstrap.cc:492] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178: No bootstrap required, opened a new log
I20260811 11:15:16.379972 29939 ts_tablet_manager.cc:1397] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 11:15:16.380846 29941 tablet_bootstrap.cc:492] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b: No bootstrap required, opened a new log
I20260811 11:15:16.380954 29941 ts_tablet_manager.cc:1397] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 11:15:16.384447 29941 raft_consensus.cc:348] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8911a74ed247bcb87f31cf091d3178" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39757 } } peers { permanent_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36241 } } peers { permanent_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43223 } }
I20260811 11:15:16.384771 29941 raft_consensus.cc:374] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:16.384832 29941 raft_consensus.cc:729] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd7ebf1e3174204aa6c02944f64bb2b, State: Initialized, Role: FOLLOWER
I20260811 11:15:16.384770 29940 raft_consensus.cc:348] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8911a74ed247bcb87f31cf091d3178" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39757 } } peers { permanent_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36241 } } peers { permanent_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43223 } }
I20260811 11:15:16.384999 29941 consensus_queue.cc:260] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8911a74ed247bcb87f31cf091d3178" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39757 } } peers { permanent_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36241 } } peers { permanent_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43223 } }
I20260811 11:15:16.385035 29939 raft_consensus.cc:348] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8911a74ed247bcb87f31cf091d3178" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39757 } } peers { permanent_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36241 } } peers { permanent_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43223 } }
I20260811 11:15:16.385141 29940 raft_consensus.cc:374] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:16.385205 29939 raft_consensus.cc:374] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:16.385201 29940 raft_consensus.cc:729] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b6f9eddd854e4f8e0a88fc740f770d, State: Initialized, Role: FOLLOWER
I20260811 11:15:16.385251 29939 raft_consensus.cc:729] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8911a74ed247bcb87f31cf091d3178, State: Initialized, Role: FOLLOWER
I20260811 11:15:16.385392 29940 consensus_queue.cc:260] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8911a74ed247bcb87f31cf091d3178" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39757 } } peers { permanent_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36241 } } peers { permanent_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43223 } }
I20260811 11:15:16.385402 29939 consensus_queue.cc:260] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8911a74ed247bcb87f31cf091d3178" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39757 } } peers { permanent_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36241 } } peers { permanent_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43223 } }
I20260811 11:15:16.385613 29941 ts_tablet_manager.cc:1428] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 11:15:16.385792 29775 heartbeater.cc:503] Master 127.28.25.62:39857 was elected leader, sending a full tablet report...
I20260811 11:15:16.385833 29939 ts_tablet_manager.cc:1428] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 11:15:16.385922 29940 ts_tablet_manager.cc:1428] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260811 11:15:16.386016 29623 heartbeater.cc:503] Master 127.28.25.62:39857 was elected leader, sending a full tablet report...
I20260811 11:15:16.385934 29922 heartbeater.cc:503] Master 127.28.25.62:39857 was elected leader, sending a full tablet report...
I20260811 11:15:16.407666 29947 raft_consensus.cc:482] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:16.407814 29947 raft_consensus.cc:504] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8911a74ed247bcb87f31cf091d3178" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39757 } } peers { permanent_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36241 } } peers { permanent_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43223 } }
I20260811 11:15:16.408429 29947 leader_election.cc:288] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af8911a74ed247bcb87f31cf091d3178 (127.28.25.1:39757), 3dd7ebf1e3174204aa6c02944f64bb2b (127.28.25.2:36241)
I20260811 11:15:16.412783 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.408675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:38729 (local address 127.28.25.1:39757)
0811 11:15:16.408723 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.408725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.408842 (+ 117us) server_negotiation.cc:407] Connection header received
0811 11:15:16.408918 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.408919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.408935 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.409011 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:16.409345 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.410078 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.411537 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.412216 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.412299 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.412312 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.412680 (+ 368us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.412723 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:15:16.412817 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.408616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:39757 (local address 127.28.25.3:38729)
0811 11:15:16.408713 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:16.408780 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:16.408854 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:16.409019 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:16.409021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:16.409023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:16.409247 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.409250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.410162 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.410164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.411440 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.411443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.412343 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.412344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.412545 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.412550 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:15:16.412670 (+ 120us) client_negotiation.cc:240] Negotiation successful
0811 11:15:16.412720 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260811 11:15:16.413762 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.408623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:36241 (local address 127.28.25.3:52211)
0811 11:15:16.409079 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:16.409092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:16.409156 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:16.409363 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:16.409364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:16.409366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:16.409628 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.409631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.410574 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.410576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.412321 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:16.412324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.413389 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.413392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:16.413559 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.413563 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:16.413617 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 11:15:16.413648 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":35,"thread_start_us":239,"threads_started":1}
I20260811 11:15:16.413772 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.408714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:52211 (local address 127.28.25.2:36241)
0811 11:15:16.408782 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.408784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.409175 (+ 391us) server_negotiation.cc:407] Connection header received
0811 11:15:16.409231 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.409232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.409248 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.409309 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:16.409758 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.410443 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.412428 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.413167 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.413248 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.413257 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.413648 (+ 391us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.413697 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:15:16.413705 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9b424eaf0a46318e46bd38300ca683" candidate_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8911a74ed247bcb87f31cf091d3178" is_pre_election: true
I20260811 11:15:16.414019 29573 raft_consensus.cc:2454] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b6f9eddd854e4f8e0a88fc740f770d in term 0.
I20260811 11:15:16.414371 29796 leader_election.cc:302] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b6f9eddd854e4f8e0a88fc740f770d, af8911a74ed247bcb87f31cf091d3178; no voters:
I20260811 11:15:16.414465 29723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9b424eaf0a46318e46bd38300ca683" candidate_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" is_pre_election: true
I20260811 11:15:16.414551 29947 raft_consensus.cc:2781] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:16.414613 29947 raft_consensus.cc:482] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:16.414654 29947 raft_consensus.cc:3037] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:16.414661 29723 raft_consensus.cc:2454] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b6f9eddd854e4f8e0a88fc740f770d in term 0.
I20260811 11:15:16.416034 29947 raft_consensus.cc:504] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8911a74ed247bcb87f31cf091d3178" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39757 } } peers { permanent_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36241 } } peers { permanent_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43223 } }
I20260811 11:15:16.416213 29947 leader_election.cc:288] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [CANDIDATE]: Term 1 election: Requested vote from peers af8911a74ed247bcb87f31cf091d3178 (127.28.25.1:39757), 3dd7ebf1e3174204aa6c02944f64bb2b (127.28.25.2:36241)
I20260811 11:15:16.416383 29723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9b424eaf0a46318e46bd38300ca683" candidate_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b"
I20260811 11:15:16.416447 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9b424eaf0a46318e46bd38300ca683" candidate_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8911a74ed247bcb87f31cf091d3178"
I20260811 11:15:16.416522 29573 raft_consensus.cc:3037] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:16.416498 29723 raft_consensus.cc:3037] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:16.417845 29573 raft_consensus.cc:2454] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b6f9eddd854e4f8e0a88fc740f770d in term 1.
I20260811 11:15:16.418054 29796 leader_election.cc:302] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b6f9eddd854e4f8e0a88fc740f770d, af8911a74ed247bcb87f31cf091d3178; no voters:
I20260811 11:15:16.418174 29947 raft_consensus.cc:2781] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:16.418217 29723 raft_consensus.cc:2454] T 1e9b424eaf0a46318e46bd38300ca683 P 3dd7ebf1e3174204aa6c02944f64bb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b6f9eddd854e4f8e0a88fc740f770d in term 1.
I20260811 11:15:16.418373 29947 raft_consensus.cc:686] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [term 1 LEADER]: Becoming Leader. State: Replica: 67b6f9eddd854e4f8e0a88fc740f770d, State: Running, Role: LEADER
I20260811 11:15:16.418497 29947 consensus_queue.cc:237] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8911a74ed247bcb87f31cf091d3178" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39757 } } peers { permanent_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36241 } } peers { permanent_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43223 } }
I20260811 11:15:16.419665 29408 catalog_manager.cc:5495] T 1e9b424eaf0a46318e46bd38300ca683 P 67b6f9eddd854e4f8e0a88fc740f770d reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b6f9eddd854e4f8e0a88fc740f770d (127.28.25.3). New cstate: current_term: 1 leader_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8911a74ed247bcb87f31cf091d3178" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd7ebf1e3174204aa6c02944f64bb2b" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67b6f9eddd854e4f8e0a88fc740f770d" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43223 } health_report { overall_health: HEALTHY } } }
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
W20260811 11:15:16.428160 29624 tablet.cc:2369] T 1e9b424eaf0a46318e46bd38300ca683 P af8911a74ed247bcb87f31cf091d3178: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:15:16.432102 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.422346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.28.25.1:39757)
0811 11:15:16.422418 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.422420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.422748 (+ 328us) server_negotiation.cc:407] Connection header received
0811 11:15:16.422822 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.422823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.422835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.422913 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:16.423723 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.424223 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.425206 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.425441 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.425521 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.428642 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:16.428653 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:16.428653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:16.428662 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:16.430735 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.431268 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.431272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.431272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.431341 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.431569 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.431570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.431570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.431724 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:16.431829 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.431951 (+ 122us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.432028 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446916, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446916, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
I20260811 11:15:16.462852 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.454445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.28.25.1:39757)
0811 11:15:16.454550 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.454553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.454676 (+ 123us) server_negotiation.cc:407] Connection header received
0811 11:15:16.454837 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.454839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.454849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.454921 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:16.455662 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.456161 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.456949 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.457156 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.457237 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.460137 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:16.460139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:16.460139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:16.460147 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:16.461580 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.461970 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.461971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.461971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.462034 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.462304 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.462305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.462306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.462429 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:16.462517 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.462696 (+ 179us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.462779 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
I20260811 11:15:16.491641 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:16.483174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.28.25.1:39757)
0811 11:15:16.483268 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:16.483270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:16.483354 (+ 84us) server_negotiation.cc:407] Connection header received
0811 11:15:16.483403 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:16.483404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:16.483414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:16.483481 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:16.484063 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.484564 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.485376 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:16.485680 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:16.485743 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:16.488833 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:16.488834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:16.488834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:16.488843 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:16.490331 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.490702 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.490703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.490704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.490762 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:16.491063 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:16.491064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:16.491065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:16.491193 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:16.491320 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:16.491443 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 11:15:16.491525 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260811 11:15:16.492470 29526 tablet_service.cc:2908] Rejecting scan request for tablet 1e9b424eaf0a46318e46bd38300ca683: Uninitialized: clean time has not yet been initialized
I20260811 11:15:16.493784 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 29480
I20260811 11:15:16.504279 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 29628
I20260811 11:15:16.513502 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 29780
I20260811 11:15:16.520766 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 29375
2026-08-11T11:15:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2233 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29967](info): set up 2 sockets
Aug 11 11:15:16 dist-test-slave-2w3w krb5kdc[29967](info): commencing operation
krb5kdc: starting...
W20260811 11:15:17.662660 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:17 dist-test-slave-2w3w krb5kdc[29967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:15:17.802567 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:45785
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:45785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:17.891569 29981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:17.891811 29981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:17.891836 29981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:17.893036 29981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:17.893076 29981 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 11:15:17.893090 29981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:17.893103 29981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:17.893115 29981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:17.893127 29981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:17.894645 29981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:45785
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:17.894901 29981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:17.895195 29981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:17.900563 29986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:17.900642 29981 server_base.cc:1034] running on GCE node
W20260811 11:15:17.900588 29989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:17.900588 29987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:17.901074 29981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:17.901433 29981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:17.902603 29981 hybrid_clock.cc:648] HybridClock initialized: now 1786446917902578 us; error 41 us; skew 500 ppm
Aug 11 11:15:17 dist-test-slave-2w3w krb5kdc[29967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446917, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:17.905539 29981 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:15:17.906071 29981 webserver.cc:460] Webserver started at http://127.28.25.62:35189/ using document root <none> and password file <none>
I20260811 11:15:17.906328 29981 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:17.906400 29981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:17.906563 29981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:17.907842 29981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "7f9c0c366ee74a7ab229b324642e7522"
format_stamp: "Formatted at 2026-08-11 11:15:17 on dist-test-slave-2w3w"
server_key: "ce3d23b01c478def39213c78e3625997"
server_key_iv: "f15e580197f96cc81c357ab2360e64cf"
server_key_version: "encryptionkey@0"
I20260811 11:15:17.908253 29981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "7f9c0c366ee74a7ab229b324642e7522"
format_stamp: "Formatted at 2026-08-11 11:15:17 on dist-test-slave-2w3w"
server_key: "ce3d23b01c478def39213c78e3625997"
server_key_iv: "f15e580197f96cc81c357ab2360e64cf"
server_key_version: "encryptionkey@0"
I20260811 11:15:17.909719 29981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 11:15:17.910568 29997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:17.910766 29981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:17.910881 29981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "7f9c0c366ee74a7ab229b324642e7522"
format_stamp: "Formatted at 2026-08-11 11:15:17 on dist-test-slave-2w3w"
server_key: "ce3d23b01c478def39213c78e3625997"
server_key_iv: "f15e580197f96cc81c357ab2360e64cf"
server_key_version: "encryptionkey@0"
I20260811 11:15:17.911074 29981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:17.929989 29981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:17.932518 29981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:17.932731 29981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:17.938155 29981 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:45785
I20260811 11:15:17.938218 30064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:45785 every 8 connection(s)
I20260811 11:15:17.938634 29981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:17.939133 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 29981
I20260811 11:15:17.939244 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:17.939430 28772 external_mini_cluster.cc:1428] Setting key e417099a366da7c5130b1652c94873bd
I20260811 11:15:17.939646 30065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:17.943389 30065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522: Bootstrap starting.
I20260811 11:15:17.944346 30065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:17.944712 30065 log.cc:824] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:17.945551 30065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522: No bootstrap required, opened a new log
Aug 11 11:15:17 dist-test-slave-2w3w krb5kdc[29967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:17.948097 30065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9c0c366ee74a7ab229b324642e7522" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45785 } }
I20260811 11:15:17.948314 30065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:17.948367 30065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f9c0c366ee74a7ab229b324642e7522, State: Initialized, Role: FOLLOWER
I20260811 11:15:17.948531 30065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9c0c366ee74a7ab229b324642e7522" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45785 } }
I20260811 11:15:17.948622 30065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:17.948648 30065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:17.948726 30065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:17.949405 30065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9c0c366ee74a7ab229b324642e7522" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45785 } }
I20260811 11:15:17.949530 30065 leader_election.cc:302] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f9c0c366ee74a7ab229b324642e7522; no voters:
I20260811 11:15:17.949751 30065 leader_election.cc:288] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:17.949848 30070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:17.950016 30070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [term 1 LEADER]: Becoming Leader. State: Replica: 7f9c0c366ee74a7ab229b324642e7522, State: Running, Role: LEADER
I20260811 11:15:17.950062 30065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:17.950152 30070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9c0c366ee74a7ab229b324642e7522" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45785 } }
I20260811 11:15:17.950516 30070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f9c0c366ee74a7ab229b324642e7522. Latest consensus state: current_term: 1 leader_uuid: "7f9c0c366ee74a7ab229b324642e7522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9c0c366ee74a7ab229b324642e7522" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45785 } } }
I20260811 11:15:17.950608 30070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:17.950563 30071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f9c0c366ee74a7ab229b324642e7522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9c0c366ee74a7ab229b324642e7522" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45785 } } }
I20260811 11:15:17.950819 30071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:17.951472 30078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:17.951754 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:17.940037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.28.25.62:45785)
0811 11:15:17.940224 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:17.940235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:17.940274 (+ 39us) server_negotiation.cc:407] Connection header received
0811 11:15:17.940865 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:17.940873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:17.940933 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:17.941059 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:17.941946 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:17.942867 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:17.943983 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:17.944501 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:17.944634 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:17.947360 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:17.947365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:17.947365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:17.947378 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:17.949306 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:17.949785 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:17.949789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:17.949790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:17.949917 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:17.950173 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:17.950174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:17.950174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:17.950527 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:17.951150 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:17.951229 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 11:15:17.951390 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260811 11:15:17.952204 30078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:17.953951 30078 catalog_manager.cc:1347] Generated new cluster ID: eb504bac9c8d4c04a8e1a7710dc35ae2
I20260811 11:15:17.954010 30078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:17.964638 30078 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:17.965265 30078 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:17.978155 30078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522: Generated new TSK 0
I20260811 11:15:17.978405 30078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:15:18.015492 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:45785
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:18.102900 30091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:18.103180 30091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:18.103209 30091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:18.104426 30091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:18.104471 30091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:18.104493 30091 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:15:18.104511 30091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:18.106036 30091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.25.62:45785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:18.106251 30091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:18.106462 30091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:18.112291 30097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:18.112280 30099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:18.112280 30096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:18.112741 30091 server_base.cc:1034] running on GCE node
I20260811 11:15:18.113005 30091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:18.113231 30091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:18.114372 30091 hybrid_clock.cc:648] HybridClock initialized: now 1786446918114353 us; error 43 us; skew 500 ppm
Aug 11 11:15:18 dist-test-slave-2w3w krb5kdc[29967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446918, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:18.117338 30091 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:15:18.117835 30091 webserver.cc:460] Webserver started at http://127.28.25.1:35463/ using document root <none> and password file <none>
I20260811 11:15:18.118088 30091 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:18.118158 30091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:18.118319 30091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:18.119545 30091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "a5fa68abeee44785bd00f86e31892712"
format_stamp: "Formatted at 2026-08-11 11:15:18 on dist-test-slave-2w3w"
server_key: "e78823585b259968db0b296adf6b8687"
server_key_iv: "8861e287e3ee5804a0c9ed9f6b5c4e2d"
server_key_version: "encryptionkey@0"
I20260811 11:15:18.119947 30091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "a5fa68abeee44785bd00f86e31892712"
format_stamp: "Formatted at 2026-08-11 11:15:18 on dist-test-slave-2w3w"
server_key: "e78823585b259968db0b296adf6b8687"
server_key_iv: "8861e287e3ee5804a0c9ed9f6b5c4e2d"
server_key_version: "encryptionkey@0"
I20260811 11:15:18.121407 30091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:15:18.122208 30109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:18.122399 30091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:18.122511 30091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "a5fa68abeee44785bd00f86e31892712"
format_stamp: "Formatted at 2026-08-11 11:15:18 on dist-test-slave-2w3w"
server_key: "e78823585b259968db0b296adf6b8687"
server_key_iv: "8861e287e3ee5804a0c9ed9f6b5c4e2d"
server_key_version: "encryptionkey@0"
I20260811 11:15:18.122593 30091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:18.145728 30091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:18.148409 30091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:18.148597 30091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:18.149188 30091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:18.149257 30091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:18.149304 30091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:18.149335 30091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:15:18 dist-test-slave-2w3w krb5kdc[29967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446918, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:18.157670 30091 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:42311
I20260811 11:15:18.157765 30237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:42311 every 8 connection(s)
I20260811 11:15:18.158157 30091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:18.160881 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.149839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:45139 (local address 127.28.25.62:45785)
0811 11:15:18.149901 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.149904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.150157 (+ 253us) server_negotiation.cc:407] Connection header received
0811 11:15:18.150835 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.150836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.150850 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.150895 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:18.151825 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.152697 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.153958 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.154209 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.154303 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.157682 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:18.157684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:18.157685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:18.157694 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:18.159358 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.159834 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.159835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.159836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.159903 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.160163 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.160164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.160164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.160298 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:18.160378 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.160677 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.160788 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 11:15:18.161087 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.149804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:45785 (local address 127.28.25.1:45139)
0811 11:15:18.150013 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.150053 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.150696 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.151038 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.151042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.151426 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:18.151646 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.151651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.152824 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.152825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.153833 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.153836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.154320 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.154320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.154472 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.155125 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:18.155134 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:18.157559 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:18.159479 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.159488 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.159491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.159737 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.159974 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.159974 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.159975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.160092 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.160368 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:18.160375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:18.160531 (+ 156us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.160642 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.160823 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260811 11:15:18.161684 30238 heartbeater.cc:347] Connected to a master server at 127.28.25.62:45785
I20260811 11:15:18.161777 30238 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:18.161995 30238 heartbeater.cc:511] Master 127.28.25.62:45785 requested a full tablet report, sending...
I20260811 11:15:18.162547 30016 ts_manager.cc:194] Registered new tserver with Master: a5fa68abeee44785bd00f86e31892712 (127.28.25.1:42311)
I20260811 11:15:18.162889 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 30091
I20260811 11:15:18.163025 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:18.163232 28772 external_mini_cluster.cc:1428] Setting key cda20972710fb342f1210340f541acad
I20260811 11:15:18.163407 30016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:45139
I20260811 11:15:18.169685 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.163550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:45785 (local address 127.28.25.1:60033)
0811 11:15:18.163868 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.163882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.163949 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.164076 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.164077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.164282 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:18.164338 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.164340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.165032 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.165033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.165571 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.165572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.165977 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.165978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.166065 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.166583 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:18.166588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:18.166893 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:18.168540 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.168542 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.168542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.168731 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.168974 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.168975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.168975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.169046 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.169316 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:18.169317 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:18.169369 (+ 52us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.169459 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.169581 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":88,"threads_started":1}
I20260811 11:15:18.169680 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.163720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:60033 (local address 127.28.25.62:45785)
0811 11:15:18.163759 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.163761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.163934 (+ 173us) server_negotiation.cc:407] Connection header received
0811 11:15:18.164008 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.164009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.164020 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.164072 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:18.164430 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.164922 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.165676 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.165898 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.165979 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.167014 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:18.167016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:18.167016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:18.167024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:18.168444 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.168810 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.168811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.168811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.168877 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.169118 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.169119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.169120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.169239 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:18.169315 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.169485 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.169579 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:15:18.194761 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:45785
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:18.284797 30249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:18.285041 30249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:18.285070 30249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:18.286307 30249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:18.286356 30249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:18.286378 30249 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:15:18.286397 30249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:18.287952 30249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.25.62:45785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:18.288192 30249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:18.288398 30249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:18.293828 30257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:18.293849 30254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:18.293978 30255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:18.294409 30249 server_base.cc:1034] running on GCE node
I20260811 11:15:18.294656 30249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:18.295023 30249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:18.296207 30249 hybrid_clock.cc:648] HybridClock initialized: now 1786446918296194 us; error 32 us; skew 500 ppm
Aug 11 11:15:18 dist-test-slave-2w3w krb5kdc[29967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446918, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:18.299288 30249 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:15:18.299779 30249 webserver.cc:460] Webserver started at http://127.28.25.2:39073/ using document root <none> and password file <none>
I20260811 11:15:18.300020 30249 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:18.300066 30249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:18.300171 30249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:18.301420 30249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "0d42301434a5430f909e138e353dcdc5"
format_stamp: "Formatted at 2026-08-11 11:15:18 on dist-test-slave-2w3w"
server_key: "37ba743a96dfb8b2412d2eaf4743ffb5"
server_key_iv: "e9f45344c55fd2b57a02df264214b599"
server_key_version: "encryptionkey@0"
I20260811 11:15:18.301764 30249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "0d42301434a5430f909e138e353dcdc5"
format_stamp: "Formatted at 2026-08-11 11:15:18 on dist-test-slave-2w3w"
server_key: "37ba743a96dfb8b2412d2eaf4743ffb5"
server_key_iv: "e9f45344c55fd2b57a02df264214b599"
server_key_version: "encryptionkey@0"
I20260811 11:15:18.303435 30249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260811 11:15:18.304420 30264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:18.304679 30249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:18.304867 30249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "0d42301434a5430f909e138e353dcdc5"
format_stamp: "Formatted at 2026-08-11 11:15:18 on dist-test-slave-2w3w"
server_key: "37ba743a96dfb8b2412d2eaf4743ffb5"
server_key_iv: "e9f45344c55fd2b57a02df264214b599"
server_key_version: "encryptionkey@0"
I20260811 11:15:18.304968 30249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:18.339905 30249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:18.342144 30249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:18.342337 30249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:18.342908 30249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:18.343004 30249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:18.343081 30249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:18.343144 30249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:15:18 dist-test-slave-2w3w krb5kdc[29967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446918, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:18.351683 30249 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:46527
I20260811 11:15:18.351786 30397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:46527 every 8 connection(s)
I20260811 11:15:18.352151 30249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:15:18.354624 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.343527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:44759 (local address 127.28.25.62:45785)
0811 11:15:18.343606 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.343609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.343853 (+ 244us) server_negotiation.cc:407] Connection header received
0811 11:15:18.344490 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.344492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.344508 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.344590 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:18.345358 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.346248 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.347960 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.348266 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.348337 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.351547 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:18.351549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:18.351550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:18.351560 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:18.353339 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.353752 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.353753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.353753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.353806 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.354047 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.354047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.354048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.354161 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:18.354213 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.354481 (+ 268us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.354550 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:15:18.354866 30274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.343515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:45785 (local address 127.28.25.2:44759)
0811 11:15:18.343707 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.343751 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.344367 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.344693 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.344697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.345029 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:18.345235 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.345239 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.346374 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.346375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.347846 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.347851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.348448 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.348450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.348606 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.349315 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:18.349326 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:18.351425 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:18.353429 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.353437 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.353438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.353676 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.353867 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.353868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.353868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.353988 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.354219 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:18.354225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:18.354352 (+ 127us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.354453 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.354589 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
I20260811 11:15:18.355460 30398 heartbeater.cc:347] Connected to a master server at 127.28.25.62:45785
I20260811 11:15:18.355551 30398 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:18.355752 30398 heartbeater.cc:511] Master 127.28.25.62:45785 requested a full tablet report, sending...
I20260811 11:15:18.356200 30017 ts_manager.cc:194] Registered new tserver with Master: 0d42301434a5430f909e138e353dcdc5 (127.28.25.2:46527)
I20260811 11:15:18.356721 30017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:44759
I20260811 11:15:18.361599 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 30249
I20260811 11:15:18.361754 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:18.361974 28772 external_mini_cluster.cc:1428] Setting key 1d905e10bcf592986b0704856d69d59f
I20260811 11:15:18.363006 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.356687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:51595 (local address 127.28.25.62:45785)
0811 11:15:18.356733 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.356736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.356834 (+ 98us) server_negotiation.cc:407] Connection header received
0811 11:15:18.356878 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.356879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.356896 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.356962 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:18.357486 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.358016 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.358736 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.358947 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.359060 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.360024 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:18.360025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:18.360025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:18.360032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:18.361261 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.361760 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.361762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.361763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.361902 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.362265 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.362266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.362266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.362470 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:18.362534 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.362739 (+ 205us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.362837 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:15:18.363020 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.356617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:45785 (local address 127.28.25.2:51595)
0811 11:15:18.356750 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.356760 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.356821 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.356969 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.356970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.357147 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:18.357210 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.357212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.358133 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.358134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.358648 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.358649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.359062 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.359063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.359144 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.359572 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:18.359576 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:18.359913 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:18.361386 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.361387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.361388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.361651 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.361965 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.361966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.361966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.362043 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.362551 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:18.362553 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:18.362608 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.362704 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.362883 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:15:18.394142 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:45785
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:18.484753 30408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:18.485030 30408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:18.485062 30408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:18.486538 30408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:18.486611 30408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:18.486637 30408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:15:18.486660 30408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:18.489001 30408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.25.62:45785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:18.489372 30408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:18.489655 30408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:18.495584 30416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:18.495643 30418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:18.495627 30415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:18.495760 30408 server_base.cc:1034] running on GCE node
I20260811 11:15:18.496055 30408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:18.496326 30408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:18.497500 30408 hybrid_clock.cc:648] HybridClock initialized: now 1786446918497483 us; error 28 us; skew 500 ppm
Aug 11 11:15:18 dist-test-slave-2w3w krb5kdc[29967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446918, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:18.506105 30408 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:15:18.506716 30408 webserver.cc:460] Webserver started at http://127.28.25.3:32795/ using document root <none> and password file <none>
I20260811 11:15:18.507069 30408 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:18.507148 30408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:18.507324 30408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:18.508610 30408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "f01cd10bf3f8490aad393439c571b22b"
format_stamp: "Formatted at 2026-08-11 11:15:18 on dist-test-slave-2w3w"
server_key: "c755ac3b37a541682c92f216840e90e4"
server_key_iv: "f274abfcacc1243cb233f37139d76ec3"
server_key_version: "encryptionkey@0"
I20260811 11:15:18.509007 30408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "f01cd10bf3f8490aad393439c571b22b"
format_stamp: "Formatted at 2026-08-11 11:15:18 on dist-test-slave-2w3w"
server_key: "c755ac3b37a541682c92f216840e90e4"
server_key_iv: "f274abfcacc1243cb233f37139d76ec3"
server_key_version: "encryptionkey@0"
I20260811 11:15:18.510749 30408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:15:18.511723 30426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:18.511971 30408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:18.512075 30408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "f01cd10bf3f8490aad393439c571b22b"
format_stamp: "Formatted at 2026-08-11 11:15:18 on dist-test-slave-2w3w"
server_key: "c755ac3b37a541682c92f216840e90e4"
server_key_iv: "f274abfcacc1243cb233f37139d76ec3"
server_key_version: "encryptionkey@0"
I20260811 11:15:18.512162 30408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:18.522075 30408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:18.524596 30408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:18.524778 30408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:18.525475 30408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:18.525521 30408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:18.525544 30408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:18.525557 30408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:15:18 dist-test-slave-2w3w krb5kdc[29967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446918, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:18.536101 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.525916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:53795 (local address 127.28.25.62:45785)
0811 11:15:18.526012 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.526015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.526305 (+ 290us) server_negotiation.cc:407] Connection header received
0811 11:15:18.526827 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.526830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.526845 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.526939 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:18.527716 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.528247 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.529335 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.529537 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.529616 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.532181 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:18.532183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:18.532183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:18.532192 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:18.533730 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.534446 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.534448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.534448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.534524 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.535118 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.535120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.535120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.535333 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:18.535414 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.535831 (+ 417us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.535973 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 11:15:18.536164 30408 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:33809
I20260811 11:15:18.536342 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.525885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:45785 (local address 127.28.25.3:53795)
0811 11:15:18.526152 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.526199 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.526725 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.527088 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.527092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.527356 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:18.527603 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.527608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.528374 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.528376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.529225 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.529228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.529628 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.529630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.529765 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.530503 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:18.530515 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:18.532065 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:18.533823 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.533830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.533834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.534336 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.534605 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.534606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.534606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.534998 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.535448 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:18.535453 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:18.535630 (+ 177us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.535751 (+ 121us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.536022 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":145,"threads_started":1}
I20260811 11:15:18.536677 30408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:15:18.537174 30558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:33809 every 8 connection(s)
I20260811 11:15:18.537364 30559 heartbeater.cc:347] Connected to a master server at 127.28.25.62:45785
I20260811 11:15:18.538086 30559 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:18.538293 30559 heartbeater.cc:511] Master 127.28.25.62:45785 requested a full tablet report, sending...
I20260811 11:15:18.538691 30017 ts_manager.cc:194] Registered new tserver with Master: f01cd10bf3f8490aad393439c571b22b (127.28.25.3:33809)
I20260811 11:15:18.539234 30017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:53795
I20260811 11:15:18.540726 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 30408
I20260811 11:15:18.540825 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:15:18.541010 28772 external_mini_cluster.cc:1428] Setting key ed7f86111d8f6b4206b8d83cae24bace
I20260811 11:15:18.541937 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:15:18.544667 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.538079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:45785 (local address 127.28.25.3:37011)
0811 11:15:18.538254 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.538269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.538331 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.538499 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.538500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.538736 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:18.538816 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.538819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.539652 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.539653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.540180 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.540182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.540546 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.540547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.540626 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.541211 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:18.541216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:18.541647 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:18.543491 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.543493 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.543493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.543749 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.544006 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.544007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.544007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.544083 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.544389 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:18.544390 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:18.544438 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.544496 (+ 58us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.544587 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260811 11:15:18.544761 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.538290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:37011 (local address 127.28.25.62:45785)
0811 11:15:18.538356 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.538358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.538385 (+ 27us) server_negotiation.cc:407] Connection header received
0811 11:15:18.538406 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.538407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.538417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.538469 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:18.538899 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.539532 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.540261 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.540447 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.540505 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.541730 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:18.541731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:18.541731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:18.541737 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:18.543406 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.543827 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.543827 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.543828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.543882 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.544150 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.544151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.544152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.544321 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:18.544395 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.544553 (+ 158us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.544642 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Aug 11 11:15:18 dist-test-slave-2w3w krb5kdc[29967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 11:15:18 dist-test-slave-2w3w krb5kdc[29967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:18.569391 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.561581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.28.25.62:45785)
0811 11:15:18.561650 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.561652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.561686 (+ 34us) server_negotiation.cc:407] Connection header received
0811 11:15:18.561717 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.561718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.561729 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.561789 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:18.562546 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.563145 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.563901 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.564127 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.564202 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.566648 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:18.566650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:18.566651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:18.566660 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:18.568136 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.568547 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.568549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.568549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.568615 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.568828 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.568829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.568830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.568960 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:18.569073 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.569190 (+ 117us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.569281 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:15:18.571681 30017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:15:18.573347 30017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:18.580677 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.576151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:42311 (local address 127.0.0.1:41234)
0811 11:15:18.576358 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.576414 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.576489 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.576956 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.576959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.576961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:18.577043 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.577045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.578123 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.578124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.579430 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.579433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.580295 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.580295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.580386 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.580401 (+ 15us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.580509 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.580573 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260811 11:15:18.580817 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.576265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.28.25.2:46527)
0811 11:15:18.576526 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.576531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.576666 (+ 135us) server_negotiation.cc:407] Connection header received
0811 11:15:18.576721 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.576724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.576744 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.576828 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:18.577156 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.578373 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.579688 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.580414 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.580497 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.580557 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.580659 (+ 102us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.580700 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":134,"threads_started":1}
I20260811 11:15:18.581112 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.576270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.28.25.1:42311)
0811 11:15:18.576789 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.576794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.576804 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:15:18.576844 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.576846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.576859 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.576949 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:18.577146 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.578037 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.579529 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.580204 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.580295 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.580371 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.580943 (+ 572us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.580992 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":105,"threads_started":1}
I20260811 11:15:18.581169 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.576613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.28.25.3:33809)
0811 11:15:18.577200 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.577205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.577216 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:15:18.577256 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.577258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.577274 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.577470 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:18.577633 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.578635 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.579926 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.580610 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.580678 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.580728 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.581020 (+ 292us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.581068 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":207,"threads_started":1}
I20260811 11:15:18.580776 30583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.576155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:46527 (local address 127.0.0.1:34384)
0811 11:15:18.576512 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.576524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.576590 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.576972 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.576973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.576976 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:18.577064 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.577067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.578467 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.578468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.579606 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.579608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.580525 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.580526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.580582 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.580585 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.580631 (+ 46us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.580664 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":101,"threads_started":1}
I20260811 11:15:18.581115 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.576399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:33809 (local address 127.0.0.1:43942)
0811 11:15:18.576615 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.576624 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.576667 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.577482 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.577483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.577485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:18.577548 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.577549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.578722 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.578723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.579815 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.579818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.580765 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.580766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.580849 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.580853 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.580921 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.580971 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":65,"threads_started":1}
I20260811 11:15:18.582019 30164 tablet_service.cc:1467] Processing CreateTablet for tablet 3c085aea92a84703a96e1a79706cd326 (DEFAULT_TABLE table=test-table [id=82b9933be44045be9c12e9bdf80ad238]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:18.582441 30164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c085aea92a84703a96e1a79706cd326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:18.582441 30481 tablet_service.cc:1467] Processing CreateTablet for tablet 3c085aea92a84703a96e1a79706cd326 (DEFAULT_TABLE table=test-table [id=82b9933be44045be9c12e9bdf80ad238]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:18.582438 30316 tablet_service.cc:1467] Processing CreateTablet for tablet 3c085aea92a84703a96e1a79706cd326 (DEFAULT_TABLE table=test-table [id=82b9933be44045be9c12e9bdf80ad238]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:18.582803 30481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c085aea92a84703a96e1a79706cd326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:18.582803 30316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c085aea92a84703a96e1a79706cd326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:18.585675 30585 tablet_bootstrap.cc:492] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: Bootstrap starting.
I20260811 11:15:18.586229 30586 tablet_bootstrap.cc:492] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: Bootstrap starting.
I20260811 11:15:18.586670 30585 tablet_bootstrap.cc:654] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:18.586853 30587 tablet_bootstrap.cc:492] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: Bootstrap starting.
I20260811 11:15:18.587184 30585 log.cc:824] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:18.587332 30586 tablet_bootstrap.cc:654] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:18.587594 30587 tablet_bootstrap.cc:654] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:18.587694 30586 log.cc:824] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:18.587981 30587 log.cc:824] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:18.588188 30585 tablet_bootstrap.cc:492] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: No bootstrap required, opened a new log
I20260811 11:15:18.588272 30585 ts_tablet_manager.cc:1397] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260811 11:15:18.588546 30586 tablet_bootstrap.cc:492] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: No bootstrap required, opened a new log
I20260811 11:15:18.588619 30586 ts_tablet_manager.cc:1397] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:15:18.588701 30587 tablet_bootstrap.cc:492] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: No bootstrap required, opened a new log
I20260811 11:15:18.588778 30587 ts_tablet_manager.cc:1397] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260811 11:15:18.590559 30587 raft_consensus.cc:348] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.590577 30585 raft_consensus.cc:348] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.590734 30587 raft_consensus.cc:374] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:18.590752 30585 raft_consensus.cc:374] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:18.590780 30587 raft_consensus.cc:729] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d42301434a5430f909e138e353dcdc5, State: Initialized, Role: FOLLOWER
I20260811 11:15:18.590795 30585 raft_consensus.cc:729] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fa68abeee44785bd00f86e31892712, State: Initialized, Role: FOLLOWER
I20260811 11:15:18.590758 30586 raft_consensus.cc:348] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.590924 30586 raft_consensus.cc:374] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:18.590991 30586 raft_consensus.cc:729] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01cd10bf3f8490aad393439c571b22b, State: Initialized, Role: FOLLOWER
I20260811 11:15:18.590940 30587 consensus_queue.cc:260] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.590951 30585 consensus_queue.cc:260] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.591104 30586 consensus_queue.cc:260] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.591271 30587 ts_tablet_manager.cc:1428] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260811 11:15:18.591404 30398 heartbeater.cc:503] Master 127.28.25.62:45785 was elected leader, sending a full tablet report...
I20260811 11:15:18.591373 30585 ts_tablet_manager.cc:1428] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 11:15:18.591523 30238 heartbeater.cc:503] Master 127.28.25.62:45785 was elected leader, sending a full tablet report...
I20260811 11:15:18.591540 30586 ts_tablet_manager.cc:1428] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:15:18.591675 30559 heartbeater.cc:503] Master 127.28.25.62:45785 was elected leader, sending a full tablet report...
W20260811 11:15:18.603032 30399 tablet.cc:2369] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:15:18.659065 30239 tablet.cc:2369] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:15:18.670701 30591 raft_consensus.cc:482] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:18.670879 30591 raft_consensus.cc:504] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.671344 30591 leader_election.cc:288] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5fa68abeee44785bd00f86e31892712 (127.28.25.1:42311), f01cd10bf3f8490aad393439c571b22b (127.28.25.3:33809)
I20260811 11:15:18.674680 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.671790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:40411 (local address 127.28.25.1:42311)
0811 11:15:18.671853 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.671855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.672068 (+ 213us) server_negotiation.cc:407] Connection header received
0811 11:15:18.672207 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.672208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.672219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.672316 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:18.672495 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.672962 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.673903 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.674294 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.674381 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.674393 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.674564 (+ 171us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.674607 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 11:15:18.674861 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.671595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:42311 (local address 127.28.25.2:40411)
0811 11:15:18.671977 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.671987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.672070 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.672325 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.672326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.672328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:18.672417 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.672418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.673055 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.673056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.673809 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.673812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.674383 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.674384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.674476 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.674480 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.674556 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.674600 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":93,"thread_start_us":215,"threads_started":1}
I20260811 11:15:18.675434 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.671844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:38803 (local address 127.28.25.3:33809)
0811 11:15:18.671893 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.671895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.671905 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:15:18.671935 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.671936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.671948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.672106 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:18.672365 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.672888 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.673932 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.674306 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.674390 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.674400 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.675325 (+ 925us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.675374 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 11:15:18.675432 30274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.671595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:33809 (local address 127.28.25.2:38803)
0811 11:15:18.671668 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.671680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.671730 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.672178 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.672181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.672183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:18.672263 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.672271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.672984 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.672985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.673771 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.673773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.674389 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.674391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.675227 (+ 836us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.675231 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.675314 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.675361 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 11:15:18.675429 30185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c085aea92a84703a96e1a79706cd326" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fa68abeee44785bd00f86e31892712" is_pre_election: true
I20260811 11:15:18.675633 30185 raft_consensus.cc:2454] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 0.
I20260811 11:15:18.675873 30504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c085aea92a84703a96e1a79706cd326" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01cd10bf3f8490aad393439c571b22b" is_pre_election: true
I20260811 11:15:18.675933 30266 leader_election.cc:302] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d42301434a5430f909e138e353dcdc5, a5fa68abeee44785bd00f86e31892712; no voters:
I20260811 11:15:18.676055 30504 raft_consensus.cc:2454] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 0.
I20260811 11:15:18.676101 30591 raft_consensus.cc:2781] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:18.676213 30591 raft_consensus.cc:482] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:18.676246 30591 raft_consensus.cc:3037] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:18.677446 30591 raft_consensus.cc:504] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.677610 30591 leader_election.cc:288] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 election: Requested vote from peers a5fa68abeee44785bd00f86e31892712 (127.28.25.1:42311), f01cd10bf3f8490aad393439c571b22b (127.28.25.3:33809)
I20260811 11:15:18.677816 30504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c085aea92a84703a96e1a79706cd326" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01cd10bf3f8490aad393439c571b22b"
I20260811 11:15:18.677865 30185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c085aea92a84703a96e1a79706cd326" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fa68abeee44785bd00f86e31892712"
I20260811 11:15:18.677922 30504 raft_consensus.cc:3037] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:18.677958 30185 raft_consensus.cc:3037] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:18.678825 30185 raft_consensus.cc:2454] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 1.
I20260811 11:15:18.679051 30266 leader_election.cc:302] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d42301434a5430f909e138e353dcdc5, a5fa68abeee44785bd00f86e31892712; no voters:
I20260811 11:15:18.679170 30591 raft_consensus.cc:2781] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:18.679425 30591 raft_consensus.cc:686] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 1 LEADER]: Becoming Leader. State: Replica: 0d42301434a5430f909e138e353dcdc5, State: Running, Role: LEADER
I20260811 11:15:18.679615 30504 raft_consensus.cc:2454] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 1.
I20260811 11:15:18.679781 30591 consensus_queue.cc:237] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.681012 30017 catalog_manager.cc:5495] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d42301434a5430f909e138e353dcdc5 (127.28.25.2). New cstate: current_term: 1 leader_uuid: "0d42301434a5430f909e138e353dcdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } health_report { overall_health: UNKNOWN } } }
I20260811 11:15:18.702315 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.699590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.28.25.2:46527)
0811 11:15:18.699666 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.699668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.699677 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:18.699725 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.699726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.699748 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.699816 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:18.699948 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.700495 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.701414 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.701606 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.701659 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.701908 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:18.702010 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:18.702073 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.702148 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.702170 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:15:18.704901 30504 raft_consensus.cc:1261] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b [term 1 FOLLOWER]: Refusing update from remote peer 0d42301434a5430f909e138e353dcdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:18.704901 30185 raft_consensus.cc:1261] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712 [term 1 FOLLOWER]: Refusing update from remote peer 0d42301434a5430f909e138e353dcdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:18.705349 30591 consensus_queue.cc:1035] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:18.705574 30596 consensus_queue.cc:1035] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:18.709982 30600 mvcc.cc:205] Tried to move back new op lower bound from 7317286579011350528 to 7317286578913230848. Current Snapshot: MvccSnapshot[applied={T|T < 7317286579011350528}]
I20260811 11:15:18.710201 30599 mvcc.cc:205] Tried to move back new op lower bound from 7317286579011350528 to 7317286578913230848. Current Snapshot: MvccSnapshot[applied={T|T < 7317286579011350528}]
I20260811 11:15:18.711731 30602 mvcc.cc:205] Tried to move back new op lower bound from 7317286579011350528 to 7317286578913230848. Current Snapshot: MvccSnapshot[applied={T|T < 7317286579011350528}]
I20260811 11:15:18.715690 30017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:18.715867 30017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:18.716925 30017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522: Sending DeleteTablet for 3 replicas of tablet 3c085aea92a84703a96e1a79706cd326
I20260811 11:15:18.717633 30164 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c085aea92a84703a96e1a79706cd326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:18 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:41234
I20260811 11:15:18.717638 30481 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c085aea92a84703a96e1a79706cd326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:18 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:43942
I20260811 11:15:18.717784 30017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:15:18.718005 30017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:18.718096 30609 tablet_replica.cc:330] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: stopping tablet replica
I20260811 11:15:18.718235 30316 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c085aea92a84703a96e1a79706cd326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:18 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:34384
I20260811 11:15:18.718804 30611 tablet_replica.cc:330] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: stopping tablet replica
I20260811 11:15:18.718927 30611 raft_consensus.cc:2229] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:15:18.719110 30611 raft_consensus.cc:2258] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:18.719267 30610 tablet_replica.cc:330] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: stopping tablet replica
I20260811 11:15:18.719628 30610 raft_consensus.cc:2229] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:18.719777 30611 ts_tablet_manager.cc:1905] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:18.720105 30164 tablet_service.cc:1467] Processing CreateTablet for tablet ba88c4f97d8b46a1af54506ab87fb6fd (DEFAULT_TABLE table=test-table [id=274b572efef240338404f8382c3d931c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:18.720273 30164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba88c4f97d8b46a1af54506ab87fb6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:18.720304 30316 tablet_service.cc:1467] Processing CreateTablet for tablet ba88c4f97d8b46a1af54506ab87fb6fd (DEFAULT_TABLE table=test-table [id=274b572efef240338404f8382c3d931c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:18.720403 30316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba88c4f97d8b46a1af54506ab87fb6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:18.722374 30587 tablet_bootstrap.cc:492] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5: Bootstrap starting.
I20260811 11:15:18.722395 30585 tablet_bootstrap.cc:492] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712: Bootstrap starting.
I20260811 11:15:18.722555 30481 tablet_service.cc:1467] Processing CreateTablet for tablet ba88c4f97d8b46a1af54506ab87fb6fd (DEFAULT_TABLE table=test-table [id=274b572efef240338404f8382c3d931c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:18.723101 30611 ts_tablet_manager.cc:1918] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:15:18.723167 30481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba88c4f97d8b46a1af54506ab87fb6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:18.723224 30611 log.cc:1196] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/3c085aea92a84703a96e1a79706cd326
I20260811 11:15:18.723623 30587 tablet_bootstrap.cc:654] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:18.723647 30611 ts_tablet_manager.cc:1939] T 3c085aea92a84703a96e1a79706cd326 P 0d42301434a5430f909e138e353dcdc5: Deleting consensus metadata
I20260811 11:15:18.724025 30585 tablet_bootstrap.cc:654] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:18.724416 29998 catalog_manager.cc:4841] TS 0d42301434a5430f909e138e353dcdc5 (127.28.25.2:46527): tablet 3c085aea92a84703a96e1a79706cd326 (table test-table [id=82b9933be44045be9c12e9bdf80ad238]) successfully deleted
I20260811 11:15:18.725522 30609 raft_consensus.cc:2229] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:18.725589 30585 tablet_bootstrap.cc:492] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712: No bootstrap required, opened a new log
I20260811 11:15:18.725696 30585 ts_tablet_manager.cc:1397] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260811 11:15:18.725759 30609 raft_consensus.cc:2258] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:18.725988 30586 tablet_bootstrap.cc:492] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: Bootstrap starting.
I20260811 11:15:18.726110 30585 raft_consensus.cc:348] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.726217 30585 raft_consensus.cc:374] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:18.726243 30585 raft_consensus.cc:729] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fa68abeee44785bd00f86e31892712, State: Initialized, Role: FOLLOWER
I20260811 11:15:18.726293 30585 consensus_queue.cc:260] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.726564 30585 ts_tablet_manager.cc:1428] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 11:15:18.727391 30586 tablet_bootstrap.cc:654] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:18.727993 30609 ts_tablet_manager.cc:1905] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:18.729323 30586 tablet_bootstrap.cc:492] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: No bootstrap required, opened a new log
I20260811 11:15:18.729403 30586 ts_tablet_manager.cc:1397] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 11:15:18.729869 30586 raft_consensus.cc:348] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.730190 30586 raft_consensus.cc:374] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:18.730228 30586 raft_consensus.cc:729] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01cd10bf3f8490aad393439c571b22b, State: Initialized, Role: FOLLOWER
I20260811 11:15:18.730427 30586 consensus_queue.cc:260] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.731159 30586 ts_tablet_manager.cc:1428] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:18.719859 30610 raft_consensus.cc:2258] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:18.731545 30609 ts_tablet_manager.cc:1918] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:15:18.731688 30609 log.cc:1196] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/3c085aea92a84703a96e1a79706cd326
I20260811 11:15:18.731956 30610 ts_tablet_manager.cc:1905] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:18.732215 30609 ts_tablet_manager.cc:1939] T 3c085aea92a84703a96e1a79706cd326 P a5fa68abeee44785bd00f86e31892712: Deleting consensus metadata
I20260811 11:15:18.733235 29999 catalog_manager.cc:4841] TS a5fa68abeee44785bd00f86e31892712 (127.28.25.1:42311): tablet 3c085aea92a84703a96e1a79706cd326 (table test-table [id=82b9933be44045be9c12e9bdf80ad238]) successfully deleted
I20260811 11:15:18.734889 30610 ts_tablet_manager.cc:1918] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:15:18.735054 30610 log.cc:1196] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/3c085aea92a84703a96e1a79706cd326
I20260811 11:15:18.735340 30610 ts_tablet_manager.cc:1939] T 3c085aea92a84703a96e1a79706cd326 P f01cd10bf3f8490aad393439c571b22b: Deleting consensus metadata
I20260811 11:15:18.735838 29998 catalog_manager.cc:4841] TS f01cd10bf3f8490aad393439c571b22b (127.28.25.3:33809): tablet 3c085aea92a84703a96e1a79706cd326 (table test-table [id=82b9933be44045be9c12e9bdf80ad238]) successfully deleted
I20260811 11:15:18.739573 30587 tablet_bootstrap.cc:492] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5: No bootstrap required, opened a new log
I20260811 11:15:18.739641 30587 ts_tablet_manager.cc:1397] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20260811 11:15:18.740252 30587 raft_consensus.cc:348] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.740368 30587 raft_consensus.cc:374] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:18.740420 30587 raft_consensus.cc:729] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d42301434a5430f909e138e353dcdc5, State: Initialized, Role: FOLLOWER
I20260811 11:15:18.740479 30587 consensus_queue.cc:260] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.740612 30587 ts_tablet_manager.cc:1428] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260811 11:15:18.787413 30560 tablet.cc:2369] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:15:18.835353 30606 raft_consensus.cc:482] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:18.835521 30606 raft_consensus.cc:504] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.835757 30606 leader_election.cc:288] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5fa68abeee44785bd00f86e31892712 (127.28.25.1:42311), f01cd10bf3f8490aad393439c571b22b (127.28.25.3:33809)
I20260811 11:15:18.835984 30185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba88c4f97d8b46a1af54506ab87fb6fd" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fa68abeee44785bd00f86e31892712" is_pre_election: true
I20260811 11:15:18.836040 30504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba88c4f97d8b46a1af54506ab87fb6fd" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01cd10bf3f8490aad393439c571b22b" is_pre_election: true
I20260811 11:15:18.836103 30185 raft_consensus.cc:2454] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 0.
I20260811 11:15:18.836133 30504 raft_consensus.cc:2454] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 0.
I20260811 11:15:18.836325 30266 leader_election.cc:302] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d42301434a5430f909e138e353dcdc5, a5fa68abeee44785bd00f86e31892712; no voters:
I20260811 11:15:18.836423 30606 raft_consensus.cc:2781] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:18.836464 30606 raft_consensus.cc:482] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:18.836522 30606 raft_consensus.cc:3037] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:18.837661 30606 raft_consensus.cc:504] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.837829 30606 leader_election.cc:288] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 election: Requested vote from peers a5fa68abeee44785bd00f86e31892712 (127.28.25.1:42311), f01cd10bf3f8490aad393439c571b22b (127.28.25.3:33809)
I20260811 11:15:18.838048 30504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba88c4f97d8b46a1af54506ab87fb6fd" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01cd10bf3f8490aad393439c571b22b"
I20260811 11:15:18.838061 30185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba88c4f97d8b46a1af54506ab87fb6fd" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fa68abeee44785bd00f86e31892712"
I20260811 11:15:18.838153 30504 raft_consensus.cc:3037] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:18.838197 30185 raft_consensus.cc:3037] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:18.839179 30504 raft_consensus.cc:2454] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 1.
I20260811 11:15:18.839459 30185 raft_consensus.cc:2454] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 1.
I20260811 11:15:18.839488 30265 leader_election.cc:302] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d42301434a5430f909e138e353dcdc5, f01cd10bf3f8490aad393439c571b22b; no voters:
I20260811 11:15:18.839720 30606 raft_consensus.cc:2781] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:18.839785 30606 raft_consensus.cc:686] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 1 LEADER]: Becoming Leader. State: Replica: 0d42301434a5430f909e138e353dcdc5, State: Running, Role: LEADER
I20260811 11:15:18.839839 30606 consensus_queue.cc:237] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.840559 30017 catalog_manager.cc:5495] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d42301434a5430f909e138e353dcdc5 (127.28.25.2). New cstate: current_term: 1 leader_uuid: "0d42301434a5430f909e138e353dcdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } health_report { overall_health: UNKNOWN } } }
Aug 11 11:15:18 dist-test-slave-2w3w krb5kdc[29967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446917, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:18.887954 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.879801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:45785 (local address 127.0.0.1:51560)
0811 11:15:18.880112 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.880122 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.880193 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.880344 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.880345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.880800 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:18.880854 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.880857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.881597 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.881597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.882163 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.882164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.882619 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.882620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.882670 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.883409 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:18.883413 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:18.885094 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:18.886694 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.886702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.886704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.886916 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.887207 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:18.887208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:18.887208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:18.887295 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:18.887597 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:18.887605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:18.887657 (+ 52us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.887734 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.887872 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":2,"thread_start_us":162,"threads_started":1}
I20260811 11:15:18.888037 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.879889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.28.25.62:45785)
0811 11:15:18.879994 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.879996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.880195 (+ 199us) server_negotiation.cc:407] Connection header received
0811 11:15:18.880263 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.880264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.880274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.880324 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:18.881003 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.881513 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.882293 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.882531 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.882609 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.885221 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:18.885223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:18.885230 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:18.885238 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:18.886606 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.887049 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.887051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.887051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.887120 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:18.887386 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:18.887387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:18.887387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:18.887520 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:18.887588 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.887768 (+ 180us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.887943 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":70}
I20260811 11:15:18.890617 30017 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:51560:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 11:15:18.890934 30017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:18.892767 30481 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6cf1aef4b7465e85860b34a7c23457 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2cbbd695dfa453184fe1bbc0e9894de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:15:18.892795 30316 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6cf1aef4b7465e85860b34a7c23457 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2cbbd695dfa453184fe1bbc0e9894de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:15:18.892946 30164 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6cf1aef4b7465e85860b34a7c23457 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2cbbd695dfa453184fe1bbc0e9894de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:15:18.893019 30316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6cf1aef4b7465e85860b34a7c23457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:18.893018 30481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6cf1aef4b7465e85860b34a7c23457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:18.893100 30164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6cf1aef4b7465e85860b34a7c23457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:18.894728 30586 tablet_bootstrap.cc:492] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: Bootstrap starting.
I20260811 11:15:18.894997 30587 tablet_bootstrap.cc:492] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: Bootstrap starting.
I20260811 11:15:18.895067 30585 tablet_bootstrap.cc:492] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: Bootstrap starting.
I20260811 11:15:18.895638 30586 tablet_bootstrap.cc:654] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:18.895671 30587 tablet_bootstrap.cc:654] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:18.895771 30585 tablet_bootstrap.cc:654] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:18.896483 30585 tablet_bootstrap.cc:492] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: No bootstrap required, opened a new log
I20260811 11:15:18.896544 30585 ts_tablet_manager.cc:1397] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:18.896723 30586 tablet_bootstrap.cc:492] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: No bootstrap required, opened a new log
I20260811 11:15:18.896781 30586 ts_tablet_manager.cc:1397] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 11:15:18.896723 30585 raft_consensus.cc:348] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.896817 30585 raft_consensus.cc:374] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:18.896842 30585 raft_consensus.cc:729] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fa68abeee44785bd00f86e31892712, State: Initialized, Role: FOLLOWER
I20260811 11:15:18.896901 30585 consensus_queue.cc:260] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.896929 30586 raft_consensus.cc:348] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.897003 30586 raft_consensus.cc:374] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:18.897009 30585 ts_tablet_manager.cc:1428] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:18.897037 30586 raft_consensus.cc:729] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01cd10bf3f8490aad393439c571b22b, State: Initialized, Role: FOLLOWER
I20260811 11:15:18.897091 30586 consensus_queue.cc:260] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.897115 30592 tablet_replica.cc:439] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } } }
I20260811 11:15:18.897207 30586 ts_tablet_manager.cc:1428] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260811 11:15:18.897244 30592 tablet_replica.cc:442] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:18.897404 30594 tablet_replica.cc:439] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } } }
I20260811 11:15:18.897477 30587 tablet_bootstrap.cc:492] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: No bootstrap required, opened a new log
I20260811 11:15:18.897501 30594 tablet_replica.cc:442] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:18.897527 30587 ts_tablet_manager.cc:1397] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 11:15:18.897696 30587 raft_consensus.cc:348] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.897764 30587 raft_consensus.cc:374] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:18.897785 30587 raft_consensus.cc:729] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d42301434a5430f909e138e353dcdc5, State: Initialized, Role: FOLLOWER
I20260811 11:15:18.897835 30587 consensus_queue.cc:260] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.897948 30587 ts_tablet_manager.cc:1428] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 11:15:18.898048 30606 tablet_replica.cc:439] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } } }
I20260811 11:15:18.898111 30606 tablet_replica.cc:442] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:18.957302 30606 raft_consensus.cc:482] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:18.957417 30606 raft_consensus.cc:504] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.957638 30606 leader_election.cc:288] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5fa68abeee44785bd00f86e31892712 (127.28.25.1:42311), f01cd10bf3f8490aad393439c571b22b (127.28.25.3:33809)
I20260811 11:15:18.957912 30185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6cf1aef4b7465e85860b34a7c23457" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fa68abeee44785bd00f86e31892712" is_pre_election: true
I20260811 11:15:18.957939 30504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6cf1aef4b7465e85860b34a7c23457" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01cd10bf3f8490aad393439c571b22b" is_pre_election: true
I20260811 11:15:18.958042 30504 raft_consensus.cc:2454] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 0.
I20260811 11:15:18.958042 30185 raft_consensus.cc:2454] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 0.
I20260811 11:15:18.958225 30266 leader_election.cc:302] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d42301434a5430f909e138e353dcdc5, a5fa68abeee44785bd00f86e31892712; no voters:
I20260811 11:15:18.958389 30606 raft_consensus.cc:2781] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:18.958442 30606 raft_consensus.cc:482] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:18.958459 30606 raft_consensus.cc:3037] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:18.959535 30606 raft_consensus.cc:504] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.959724 30606 leader_election.cc:288] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 election: Requested vote from peers a5fa68abeee44785bd00f86e31892712 (127.28.25.1:42311), f01cd10bf3f8490aad393439c571b22b (127.28.25.3:33809)
I20260811 11:15:18.959921 30185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6cf1aef4b7465e85860b34a7c23457" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fa68abeee44785bd00f86e31892712"
I20260811 11:15:18.960037 30185 raft_consensus.cc:3037] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:18.960000 30504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6cf1aef4b7465e85860b34a7c23457" candidate_uuid: "0d42301434a5430f909e138e353dcdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01cd10bf3f8490aad393439c571b22b"
I20260811 11:15:18.960076 30504 raft_consensus.cc:3037] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:18.960907 30185 raft_consensus.cc:2454] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 1.
I20260811 11:15:18.960939 30504 raft_consensus.cc:2454] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d42301434a5430f909e138e353dcdc5 in term 1.
I20260811 11:15:18.961153 30265 leader_election.cc:302] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d42301434a5430f909e138e353dcdc5, f01cd10bf3f8490aad393439c571b22b; no voters:
I20260811 11:15:18.961298 30606 raft_consensus.cc:2781] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:18.961378 30606 raft_consensus.cc:686] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [term 1 LEADER]: Becoming Leader. State: Replica: 0d42301434a5430f909e138e353dcdc5, State: Running, Role: LEADER
I20260811 11:15:18.961445 30606 consensus_queue.cc:237] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } }
I20260811 11:15:18.961841 30596 tablet_replica.cc:439] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: TxnStatusTablet state changed. Reason: New leader 0d42301434a5430f909e138e353dcdc5. Latest consensus state: current_term: 1 leader_uuid: "0d42301434a5430f909e138e353dcdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } } }
I20260811 11:15:18.961947 30596 tablet_replica.cc:442] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:15:18.962107 30628 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 11:15:18.962260 30628 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 11:15:18.962199 30016 catalog_manager.cc:5495] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d42301434a5430f909e138e353dcdc5 (127.28.25.2). New cstate: current_term: 1 leader_uuid: "0d42301434a5430f909e138e353dcdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } health_report { overall_health: UNKNOWN } } }
I20260811 11:15:18.975656 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.972828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.28.25.2:46527)
0811 11:15:18.972892 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.972894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.972935 (+ 41us) server_negotiation.cc:407] Connection header received
0811 11:15:18.972962 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.972963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.972973 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.973013 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:18.973167 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.973689 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.974838 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.975129 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.975198 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.975417 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:18.975451 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:18.975493 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.975566 (+ 73us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.975590 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 11:15:18.975703 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.972701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:46527 (local address 127.0.0.1:34408)
0811 11:15:18.972807 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.972820 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.972867 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.973044 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.973045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.973046 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:15:18.973105 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.973107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.973765 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.973766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.974697 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.974699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.975212 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.975213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.975288 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.975315 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:15:18.975512 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:15:18.975518 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.975565 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.975617 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260811 11:15:18.977017 30504 raft_consensus.cc:1261] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b [term 1 FOLLOWER]: Refusing update from remote peer 0d42301434a5430f909e138e353dcdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:18.977006 30185 raft_consensus.cc:1261] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712 [term 1 FOLLOWER]: Refusing update from remote peer 0d42301434a5430f909e138e353dcdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:18.977277 30596 consensus_queue.cc:1035] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:18.977396 30606 consensus_queue.cc:1035] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:18.978029 30594 tablet_replica.cc:439] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: TxnStatusTablet state changed. Reason: New leader 0d42301434a5430f909e138e353dcdc5. Latest consensus state: current_term: 1 leader_uuid: "0d42301434a5430f909e138e353dcdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } } }
I20260811 11:15:18.978060 30592 tablet_replica.cc:439] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: TxnStatusTablet state changed. Reason: New leader 0d42301434a5430f909e138e353dcdc5. Latest consensus state: current_term: 1 leader_uuid: "0d42301434a5430f909e138e353dcdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } } }
I20260811 11:15:18.978129 30594 tablet_replica.cc:442] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:18.978142 30592 tablet_replica.cc:442] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:18.979141 30606 tablet_replica.cc:439] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d42301434a5430f909e138e353dcdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } } }
I20260811 11:15:18.979239 30606 tablet_replica.cc:442] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:15:18.979288 30606 tablet_replica.cc:439] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d42301434a5430f909e138e353dcdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } } }
I20260811 11:15:18.979352 30606 tablet_replica.cc:442] T 5a6cf1aef4b7465e85860b34a7c23457 P 0d42301434a5430f909e138e353dcdc5: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:15:18.979557 30592 tablet_replica.cc:439] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d42301434a5430f909e138e353dcdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } } }
I20260811 11:15:18.979653 30592 tablet_replica.cc:442] T 5a6cf1aef4b7465e85860b34a7c23457 P a5fa68abeee44785bd00f86e31892712: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:18.979782 30594 tablet_replica.cc:439] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d42301434a5430f909e138e353dcdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 } } peers { permanent_uuid: "0d42301434a5430f909e138e353dcdc5" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 46527 } } peers { permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 } } }
I20260811 11:15:18.979880 30594 tablet_replica.cc:442] T 5a6cf1aef4b7465e85860b34a7c23457 P f01cd10bf3f8490aad393439c571b22b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:18.987571 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.984598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:44713 (local address 127.28.25.2:46527)
0811 11:15:18.984644 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:18.984646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:18.984895 (+ 249us) server_negotiation.cc:407] Connection header received
0811 11:15:18.984921 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:18.984922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:18.984931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:18.984983 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:18.985126 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.985600 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.986744 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.987081 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.987141 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.987340 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:18.987372 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:18.987411 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:18.987457 (+ 46us) server_negotiation.cc:299] Negotiation successful
0811 11:15:18.987477 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:15:18.987682 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:18.984532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:46527 (local address 127.28.25.2:44713)
0811 11:15:18.984772 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:18.984783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:18.984852 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:18.985006 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:18.985006 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:18.985008 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:15:18.985066 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.985067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.985673 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.985674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.986642 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:18.986645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:18.987156 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:18.987158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:18.987223 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:18.987247 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:15:18.987412 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:15:18.987414 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 11:15:18.987457 (+ 43us) client_negotiation.cc:240] Negotiation successful
0811 11:15:18.987507 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260811 11:15:18.990089 30504 raft_consensus.cc:1261] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b [term 1 FOLLOWER]: Refusing update from remote peer 0d42301434a5430f909e138e353dcdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:18.990120 30185 raft_consensus.cc:1261] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712 [term 1 FOLLOWER]: Refusing update from remote peer 0d42301434a5430f909e138e353dcdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:18.990368 30591 consensus_queue.cc:1035] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fa68abeee44785bd00f86e31892712" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:18.990468 30596 consensus_queue.cc:1035] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f01cd10bf3f8490aad393439c571b22b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:19.005044 30017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:19.005177 30017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:19.005877 30017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f9c0c366ee74a7ab229b324642e7522: Sending DeleteTablet for 3 replicas of tablet ba88c4f97d8b46a1af54506ab87fb6fd
I20260811 11:15:19.006129 30316 tablet_service.cc:1514] Processing DeleteTablet for tablet ba88c4f97d8b46a1af54506ab87fb6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:19 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:34384
I20260811 11:15:19.006156 30164 tablet_service.cc:1514] Processing DeleteTablet for tablet ba88c4f97d8b46a1af54506ab87fb6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:19 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:41234
I20260811 11:15:19.006163 30481 tablet_service.cc:1514] Processing DeleteTablet for tablet ba88c4f97d8b46a1af54506ab87fb6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:19 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:43942
I20260811 11:15:19.006266 30611 tablet_replica.cc:330] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5: stopping tablet replica
I20260811 11:15:19.006269 30609 tablet_replica.cc:330] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712: stopping tablet replica
I20260811 11:15:19.006330 30609 raft_consensus.cc:2229] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:19.006369 30609 raft_consensus.cc:2258] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:19.006327 30611 raft_consensus.cc:2229] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:15:19.006479 30611 raft_consensus.cc:2258] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:19.006479 30610 tablet_replica.cc:330] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: stopping tablet replica
I20260811 11:15:19.006533 30610 raft_consensus.cc:2229] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:19.006572 30610 raft_consensus.cc:2258] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:19.006938 30611 ts_tablet_manager.cc:1905] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:19.008189 30609 ts_tablet_manager.cc:1905] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:19.009308 30611 ts_tablet_manager.cc:1918] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:15:19.009380 30611 log.cc:1196] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/ba88c4f97d8b46a1af54506ab87fb6fd
I20260811 11:15:19.009611 30611 ts_tablet_manager.cc:1939] T ba88c4f97d8b46a1af54506ab87fb6fd P 0d42301434a5430f909e138e353dcdc5: Deleting consensus metadata
I20260811 11:15:19.010008 29998 catalog_manager.cc:4841] TS 0d42301434a5430f909e138e353dcdc5 (127.28.25.2:46527): tablet ba88c4f97d8b46a1af54506ab87fb6fd (table test-table [id=274b572efef240338404f8382c3d931c]) successfully deleted
I20260811 11:15:19.010192 30610 ts_tablet_manager.cc:1905] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:19.011278 30609 ts_tablet_manager.cc:1918] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:15:19.011360 30609 log.cc:1196] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/ba88c4f97d8b46a1af54506ab87fb6fd
I20260811 11:15:19.011592 30609 ts_tablet_manager.cc:1939] T ba88c4f97d8b46a1af54506ab87fb6fd P a5fa68abeee44785bd00f86e31892712: Deleting consensus metadata
I20260811 11:15:19.012042 29999 catalog_manager.cc:4841] TS a5fa68abeee44785bd00f86e31892712 (127.28.25.1:42311): tablet ba88c4f97d8b46a1af54506ab87fb6fd (table test-table [id=274b572efef240338404f8382c3d931c]) successfully deleted
I20260811 11:15:19.012223 30610 ts_tablet_manager.cc:1918] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:15:19.012297 30610 log.cc:1196] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/ba88c4f97d8b46a1af54506ab87fb6fd
I20260811 11:15:19.012519 30610 ts_tablet_manager.cc:1939] T ba88c4f97d8b46a1af54506ab87fb6fd P f01cd10bf3f8490aad393439c571b22b: Deleting consensus metadata
I20260811 11:15:19.012893 29998 catalog_manager.cc:4841] TS f01cd10bf3f8490aad393439c571b22b (127.28.25.3:33809): tablet ba88c4f97d8b46a1af54506ab87fb6fd (table test-table [id=274b572efef240338404f8382c3d931c]) successfully deleted
Aug 11 11:15:19 dist-test-slave-2w3w krb5kdc[29967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
I20260811 11:15:19.020520 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:19.008792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.28.25.1:42311)
0811 11:15:19.008884 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:19.008886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:19.010264 (+ 1378us) server_negotiation.cc:407] Connection header received
0811 11:15:19.010393 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:19.010394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:19.010407 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:19.010455 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:19.011456 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:19.012203 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:19.013361 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:19.013643 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:19.013721 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:19.017308 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:19.017325 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:19.017326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:19.017335 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:19.019242 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:19.019645 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:19.019650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:19.019650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:19.019719 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:19.019937 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:19.019938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:19.019938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:19.020102 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:19.020267 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:19.020312 (+ 45us) server_negotiation.cc:299] Negotiation successful
0811 11:15:19.020405 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260811 11:15:19.020923 30232 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41244
I20260811 11:15:19.029281 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:19.022099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.28.25.62:45785)
0811 11:15:19.022150 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:19.022152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:19.022163 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:15:19.022244 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:19.022245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:19.022256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:19.022324 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:19.023062 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:19.023596 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:19.024623 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:19.024964 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:19.025026 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:19.026596 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:19.026597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:19.026598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:19.026604 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:19.028127 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:19.028524 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:19.028525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:19.028525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:19.028580 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:19.028780 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:19.028780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:19.028781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:19.028931 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:19.029008 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:19.029149 (+ 141us) server_negotiation.cc:299] Negotiation successful
0811 11:15:19.029211 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260811 11:15:19.029608 30017 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51562
I20260811 11:15:19.030344 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 30091
I20260811 11:15:19.039762 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 30249
I20260811 11:15:19.051509 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 30408
I20260811 11:15:19.060760 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 29981
2026-08-11T11:15:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2542 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:19 dist-test-slave-2w3w krb5kdc[30645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:19 dist-test-slave-2w3w krb5kdc[30645](info): set up 2 sockets
Aug 11 11:15:19 dist-test-slave-2w3w krb5kdc[30645](info): commencing operation
krb5kdc: starting...
W20260811 11:15:20.188231 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:20 dist-test-slave-2w3w krb5kdc[30645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:15:20.326350 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33887
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:33887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:20.417097 30660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:20.417368 30660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:20.417397 30660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:20.418788 30660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:20.418857 30660 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 11:15:20.418871 30660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:20.418885 30660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:20.418938 30660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:20.418978 30660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:20.420722 30660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:33887
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:20.421052 30660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:20.421347 30660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:20.426945 30669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:20.426918 30666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:20.427235 30660 server_base.cc:1034] running on GCE node
W20260811 11:15:20.426920 30667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:20.427575 30660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:20.427964 30660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:20.429169 30660 hybrid_clock.cc:648] HybridClock initialized: now 1786446920429137 us; error 47 us; skew 500 ppm
Aug 11 11:15:20 dist-test-slave-2w3w krb5kdc[30645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446920, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:20.432157 30660 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:15:20.432624 30660 webserver.cc:460] Webserver started at http://127.28.25.62:37523/ using document root <none> and password file <none>
I20260811 11:15:20.432905 30660 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:20.432958 30660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:20.433122 30660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:20.434278 30660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "fe4f0a88d4224922ad9268fc460c9927"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "84d43d90af16fb217e13d64d23a5a9a7"
server_key_iv: "ed02c914ddbbd490ab3978068d214525"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.434680 30660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "fe4f0a88d4224922ad9268fc460c9927"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "84d43d90af16fb217e13d64d23a5a9a7"
server_key_iv: "ed02c914ddbbd490ab3978068d214525"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.436339 30660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:15:20.437320 30677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:20.437542 30660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:20.437661 30660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "fe4f0a88d4224922ad9268fc460c9927"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "84d43d90af16fb217e13d64d23a5a9a7"
server_key_iv: "ed02c914ddbbd490ab3978068d214525"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.437744 30660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:20.456223 30660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:20.458810 30660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:20.459020 30660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:20.464447 30660 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:33887
I20260811 11:15:20.464552 30747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:33887 every 8 connection(s)
I20260811 11:15:20.464938 30660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:20.465710 30748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:20.468349 30748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927: Bootstrap starting.
I20260811 11:15:20.469012 30748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:20.469342 30748 log.cc:824] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:20.470177 30748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927: No bootstrap required, opened a new log
I20260811 11:15:20.472760 30748 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4f0a88d4224922ad9268fc460c9927" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33887 } }
I20260811 11:15:20.472936 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 30660
I20260811 11:15:20.472996 30748 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:20.473029 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:20.473040 30748 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4f0a88d4224922ad9268fc460c9927, State: Initialized, Role: FOLLOWER
I20260811 11:15:20.473201 28772 external_mini_cluster.cc:1428] Setting key aefe17ba853cd10b5439fc67098f838d
I20260811 11:15:20.473212 30748 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4f0a88d4224922ad9268fc460c9927" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33887 } }
I20260811 11:15:20.473312 30748 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:20.473353 30748 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:20.473398 30748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:20.474274 30748 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4f0a88d4224922ad9268fc460c9927" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33887 } }
I20260811 11:15:20.474396 30748 leader_election.cc:302] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe4f0a88d4224922ad9268fc460c9927; no voters:
I20260811 11:15:20.474568 30748 leader_election.cc:288] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:20.474701 30753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:20.474947 30753 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [term 1 LEADER]: Becoming Leader. State: Replica: fe4f0a88d4224922ad9268fc460c9927, State: Running, Role: LEADER
I20260811 11:15:20.475083 30748 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:20.475091 30753 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4f0a88d4224922ad9268fc460c9927" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33887 } }
I20260811 11:15:20.475507 30756 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe4f0a88d4224922ad9268fc460c9927. Latest consensus state: current_term: 1 leader_uuid: "fe4f0a88d4224922ad9268fc460c9927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4f0a88d4224922ad9268fc460c9927" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33887 } } }
I20260811 11:15:20.475765 30756 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:20.475690 30754 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe4f0a88d4224922ad9268fc460c9927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4f0a88d4224922ad9268fc460c9927" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33887 } } }
I20260811 11:15:20.475943 30754 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:20.476162 30765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:20.476812 30765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:20.478933 30765 catalog_manager.cc:1347] Generated new cluster ID: 5f2007ad000942d7b07e3c34dd7c0833
I20260811 11:15:20.479029 30765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 11:15:20 dist-test-slave-2w3w krb5kdc[30645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:20.486132 30765 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:20.487066 30765 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:20.487654 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:20.473757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.28.25.62:33887)
0811 11:15:20.474088 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:20.474100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:20.474143 (+ 43us) server_negotiation.cc:407] Connection header received
0811 11:15:20.474286 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:20.474294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:20.474361 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:20.474523 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:20.475485 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.476999 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.477906 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.478542 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.478672 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:20.481687 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:20.481692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:20.481692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:20.481707 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:20.485243 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:20.485866 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:20.485871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:20.485871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:20.486030 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:20.486317 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:20.486318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:20.486318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:20.486633 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:20.486776 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:20.487208 (+ 432us) server_negotiation.cc:299] Negotiation successful
0811 11:15:20.487400 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":196,"threads_started":1}
I20260811 11:15:20.491712 30765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927: Generated new TSK 0
I20260811 11:15:20.491964 30765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:15:20.526270 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33887
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:20.622329 30777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:20.622582 30777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:20.622617 30777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:20.624078 30777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:20.624137 30777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:20.624161 30777 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 11:15:20.624174 30777 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:15:20.624220 30777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:20.626000 30777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.25.62:33887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:20.626308 30777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:20.626595 30777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:20.632634 30783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:20.632638 30785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:20.632611 30782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:20.632858 30777 server_base.cc:1034] running on GCE node
I20260811 11:15:20.633138 30777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:20.633409 30777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:20.634588 30777 hybrid_clock.cc:648] HybridClock initialized: now 1786446920634555 us; error 45 us; skew 500 ppm
Aug 11 11:15:20 dist-test-slave-2w3w krb5kdc[30645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446920, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:20.637583 30777 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:15:20.638119 30777 webserver.cc:460] Webserver started at http://127.28.25.1:41557/ using document root <none> and password file <none>
I20260811 11:15:20.638391 30777 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:20.638468 30777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:20.638628 30777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:20.639853 30777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "26abc7b9c0734f31b2bb916adc9aea1d"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "b2d1e0853acbbf5a85a9db5d1a3f6b13"
server_key_iv: "84e18b003dde4b8312c4c96cf77de57f"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.640235 30777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "26abc7b9c0734f31b2bb916adc9aea1d"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "b2d1e0853acbbf5a85a9db5d1a3f6b13"
server_key_iv: "84e18b003dde4b8312c4c96cf77de57f"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.642030 30777 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:15:20.642920 30793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:20.643175 30777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:20.643290 30777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "26abc7b9c0734f31b2bb916adc9aea1d"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "b2d1e0853acbbf5a85a9db5d1a3f6b13"
server_key_iv: "84e18b003dde4b8312c4c96cf77de57f"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.643376 30777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:20.656258 30777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:20.658895 30777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:20.659170 30777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:20.659739 30777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:20.659808 30777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:20.659860 30777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:20.659888 30777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:20.667615 30777 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:32859
I20260811 11:15:20.667677 30925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:32859 every 8 connection(s)
Aug 11 11:15:20 dist-test-slave-2w3w krb5kdc[30645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446920, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:20.668339 30777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:20.671834 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:20.660416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:57239 (local address 127.28.25.62:33887)
0811 11:15:20.660477 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:20.660480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:20.660886 (+ 406us) server_negotiation.cc:407] Connection header received
0811 11:15:20.661632 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:20.661634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:20.661651 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:20.661729 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:20.662728 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.663645 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.665439 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.665808 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.665898 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:20.668686 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:20.668688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:20.668688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:20.668698 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:20.670233 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:20.670714 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:20.670715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:20.670715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:20.670790 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:20.671095 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:20.671103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:20.671104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:20.671243 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:20.671311 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:20.671648 (+ 337us) server_negotiation.cc:299] Negotiation successful
0811 11:15:20.671741 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 11:15:20.672062 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:20.660432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33887 (local address 127.28.25.1:57239)
0811 11:15:20.660662 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:20.660704 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:20.661412 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:20.661829 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:20.661834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:20.662267 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:20.662592 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:20.662600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.663769 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.663771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:20.665318 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:20.665321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.665936 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.665938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:20.666088 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:20.666825 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:20.666834 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:20.668530 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:20.670345 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:20.670353 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:20.670354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:20.670603 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:20.670906 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:20.670907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:20.670907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:20.671038 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:20.671340 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:20.671347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:20.671487 (+ 140us) client_negotiation.cc:769] Sending connection context
0811 11:15:20.671595 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 11:15:20.671792 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":105,"threads_started":1}
I20260811 11:15:20.672619 30926 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33887
I20260811 11:15:20.672724 30926 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:20.672960 30926 heartbeater.cc:511] Master 127.28.25.62:33887 requested a full tablet report, sending...
I20260811 11:15:20.673071 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 30777
I20260811 11:15:20.673194 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:20.673445 28772 external_mini_cluster.cc:1428] Setting key 98fbcaaf10e19570af83f17730154139
I20260811 11:15:20.673619 30695 ts_manager.cc:194] Registered new tserver with Master: 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859)
I20260811 11:15:20.674640 30695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:57239
I20260811 11:15:20.681073 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:20.673923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33887 (local address 127.28.25.1:41355)
0811 11:15:20.674078 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:20.674088 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:20.674154 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:20.674288 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:20.674289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:20.674495 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:20.674569 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:20.674571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.675578 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.675579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:20.676273 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:20.676276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.676644 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.676645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:20.676731 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:20.677326 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:20.677332 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:20.677799 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:20.679829 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:20.679831 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:20.679832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:20.680062 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:20.680327 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:20.680328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:20.680328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:20.680408 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:20.680774 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:20.680776 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:20.680825 (+ 49us) client_negotiation.cc:769] Sending connection context
0811 11:15:20.680885 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 11:15:20.680989 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260811 11:15:20.681185 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:20.674023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:41355 (local address 127.28.25.62:33887)
0811 11:15:20.674064 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:20.674066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:20.674132 (+ 66us) server_negotiation.cc:407] Connection header received
0811 11:15:20.674220 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:20.674221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:20.674232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:20.674287 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:20.674654 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.675459 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.676365 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.676556 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.676624 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:20.677906 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:20.677908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:20.677908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:20.677916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:20.679694 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:20.680156 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:20.680158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:20.680158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:20.680230 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:20.680495 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:20.680496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:20.680496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:20.680670 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:20.680762 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:20.680938 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 11:15:20.681037 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:15:20.707759 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33887
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:20.798123 30936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:20.798419 30936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:20.798467 30936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:20.800199 30936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:20.800392 30936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:20.800438 30936 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 11:15:20.800469 30936 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:15:20.800504 30936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:20.802268 30936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.25.62:33887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:20.802588 30936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:20.802856 30936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:20.808708 30944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:20.808769 30946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:20.808984 30936 server_base.cc:1034] running on GCE node
W20260811 11:15:20.808708 30943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:20.809443 30936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:20.809777 30936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:20.811059 30936 hybrid_clock.cc:648] HybridClock initialized: now 1786446920811049 us; error 42 us; skew 500 ppm
Aug 11 11:15:20 dist-test-slave-2w3w krb5kdc[30645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446920, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:20.813807 30936 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:15:20.814399 30936 webserver.cc:460] Webserver started at http://127.28.25.2:36899/ using document root <none> and password file <none>
I20260811 11:15:20.814646 30936 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:20.814692 30936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:20.814903 30936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:20.816126 30936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "fd7bb18d26df4a76bb342bc5d330d60d"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "eda7f1db3d7edbb33f4140547c809c98"
server_key_iv: "dcb6d07f6a424fa0870ab05835592abd"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.816506 30936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "fd7bb18d26df4a76bb342bc5d330d60d"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "eda7f1db3d7edbb33f4140547c809c98"
server_key_iv: "dcb6d07f6a424fa0870ab05835592abd"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.818069 30936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:15:20.818894 30954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:20.819195 30936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:20.819311 30936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "fd7bb18d26df4a76bb342bc5d330d60d"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "eda7f1db3d7edbb33f4140547c809c98"
server_key_iv: "dcb6d07f6a424fa0870ab05835592abd"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.819407 30936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:20.834542 30936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:20.836942 30936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:20.837138 30936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:20.837863 30936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:20.837931 30936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:20.837970 30936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:20.837983 30936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:15:20 dist-test-slave-2w3w krb5kdc[30645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446920, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:20.846324 30936 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:34293
I20260811 11:15:20.846503 31082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:34293 every 8 connection(s)
I20260811 11:15:20.846786 30936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:15:20.849437 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:20.838396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:37177 (local address 127.28.25.62:33887)
0811 11:15:20.838460 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:20.838464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:20.838755 (+ 291us) server_negotiation.cc:407] Connection header received
0811 11:15:20.839614 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:20.839617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:20.839633 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:20.839719 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:20.840801 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.841329 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.842572 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.842819 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.842907 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:20.846099 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:20.846101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:20.846101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:20.846111 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:20.847970 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:20.848463 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:20.848464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:20.848464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:20.848531 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:20.848808 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:20.848810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:20.848810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:20.848956 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:20.849020 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:20.849282 (+ 262us) server_negotiation.cc:299] Negotiation successful
0811 11:15:20.849359 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:15:20.849648 30964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:20.838340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33887 (local address 127.28.25.2:37177)
0811 11:15:20.838581 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:20.838640 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:20.839448 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:20.839822 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:20.839827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:20.840294 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:20.840678 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:20.840687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.841465 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.841466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:20.842461 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:20.842464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.842925 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.842926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:20.843105 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:20.843714 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:20.843724 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:20.845957 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:20.848072 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:20.848079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:20.848081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:20.848341 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:20.848608 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:20.848608 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:20.848609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:20.848721 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:20.849022 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:20.849027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:20.849160 (+ 133us) client_negotiation.cc:769] Sending connection context
0811 11:15:20.849248 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 11:15:20.849417 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260811 11:15:20.850206 31083 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33887
I20260811 11:15:20.850332 31083 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:20.850558 31083 heartbeater.cc:511] Master 127.28.25.62:33887 requested a full tablet report, sending...
I20260811 11:15:20.851023 30696 ts_manager.cc:194] Registered new tserver with Master: fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293)
I20260811 11:15:20.851583 30696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:37177
I20260811 11:15:20.855263 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 30936
I20260811 11:15:20.855407 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:20.855631 28772 external_mini_cluster.cc:1428] Setting key c78ddbf11754f199156b6a7e56aab6b2
I20260811 11:15:20.858592 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:20.851501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:43931 (local address 127.28.25.62:33887)
0811 11:15:20.851549 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:20.851551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:20.851631 (+ 80us) server_negotiation.cc:407] Connection header received
0811 11:15:20.851736 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:20.851737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:20.851748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:20.851807 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:20.852245 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.852793 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.853696 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.853929 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.853991 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:20.855323 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:20.855325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:20.855325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:20.855334 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:20.857066 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:20.857586 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:20.857588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:20.857588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:20.857666 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:20.857949 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:20.857950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:20.857951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:20.858131 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:20.858218 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:20.858390 (+ 172us) server_negotiation.cc:299] Negotiation successful
0811 11:15:20.858487 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:15:20.858610 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:20.851393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33887 (local address 127.28.25.2:43931)
0811 11:15:20.851561 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:20.851573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:20.851655 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:20.851829 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:20.851830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:20.852070 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:20.852150 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:20.852152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.853020 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.853022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:20.853595 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:20.853597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:20.854013 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:20.854014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:20.854085 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:20.854696 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:20.854702 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:20.855204 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:20.857180 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:20.857182 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:20.857182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:20.857458 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:20.857741 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:20.857742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:20.857742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:20.857819 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:20.858207 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:20.858208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:20.858267 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 11:15:20.858362 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 11:15:20.858488 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:15:20.887509 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33887
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:20.978610 31093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:20.978886 31093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:20.978935 31093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:20.980280 31093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:20.980365 31093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:20.980404 31093 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 11:15:20.980453 31093 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:15:20.980494 31093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:20.982162 31093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.25.62:33887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:20.982498 31093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:20.982785 31093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:20.988612 31099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:20.988612 31101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:20.988615 31098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:20.989068 31093 server_base.cc:1034] running on GCE node
I20260811 11:15:20.989427 31093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:20.989723 31093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:20.990931 31093 hybrid_clock.cc:648] HybridClock initialized: now 1786446920990913 us; error 36 us; skew 500 ppm
Aug 11 11:15:20 dist-test-slave-2w3w krb5kdc[30645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446920, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:20.994030 31093 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:15:20.994572 31093 webserver.cc:460] Webserver started at http://127.28.25.3:45279/ using document root <none> and password file <none>
I20260811 11:15:20.994871 31093 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:20.994923 31093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:20.995132 31093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:20.996284 31093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "9040217d356e4373b2fa651d958dcc6b"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "1193037a72fa612797ed7f88beb629c6"
server_key_iv: "be49b55f7fd1195316b53160ff6d58a1"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.996673 31093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "9040217d356e4373b2fa651d958dcc6b"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "1193037a72fa612797ed7f88beb629c6"
server_key_iv: "be49b55f7fd1195316b53160ff6d58a1"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.998364 31093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:15:20.999349 31109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:20.999573 31093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:20.999691 31093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "9040217d356e4373b2fa651d958dcc6b"
format_stamp: "Formatted at 2026-08-11 11:15:20 on dist-test-slave-2w3w"
server_key: "1193037a72fa612797ed7f88beb629c6"
server_key_iv: "be49b55f7fd1195316b53160ff6d58a1"
server_key_version: "encryptionkey@0"
I20260811 11:15:20.999780 31093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:21.017140 31093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:21.019896 31093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:21.020097 31093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:21.020798 31093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:21.020854 31093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:21.020895 31093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:21.020907 31093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:21.029013 31093 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:44019
Aug 11 11:15:21 dist-test-slave-2w3w krb5kdc[30645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446920, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:21.029178 31241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:44019 every 8 connection(s)
I20260811 11:15:21.029469 31093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:15:21.032963 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.021357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:59133 (local address 127.28.25.62:33887)
0811 11:15:21.021437 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.021441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.021694 (+ 253us) server_negotiation.cc:407] Connection header received
0811 11:15:21.022241 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.022243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.022260 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.022356 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:21.023411 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.024042 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.025977 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.026344 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.026459 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.029790 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:21.029793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:21.029793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:21.029803 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:21.031480 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:21.032019 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:21.032020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:21.032021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:21.032084 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:21.032352 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:21.032353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:21.032353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:21.032476 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:21.032532 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.032800 (+ 268us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.032892 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:15:21.033274 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.021345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33887 (local address 127.28.25.3:59133)
0811 11:15:21.021541 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.021586 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.022130 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.022424 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.022428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.023003 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:21.023280 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.023288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.024159 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.024160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.025832 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.025839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.026472 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.026474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.026638 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.027365 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:21.027377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:21.029626 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:21.031583 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:21.031592 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:21.031594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:21.031928 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:21.032154 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:21.032154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:21.032155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:21.032271 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:21.032537 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:21.032542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:21.032679 (+ 137us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.032767 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.032972 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":119,"threads_started":1}
I20260811 11:15:21.033993 31242 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33887
I20260811 11:15:21.034098 31242 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:21.034313 31242 heartbeater.cc:511] Master 127.28.25.62:33887 requested a full tablet report, sending...
I20260811 11:15:21.034636 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 31093
I20260811 11:15:21.034775 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:15:21.034814 30696 ts_manager.cc:194] Registered new tserver with Master: 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019)
I20260811 11:15:21.035086 28772 external_mini_cluster.cc:1428] Setting key 3bb9295058d04b0dbdc755a2949c03ec
I20260811 11:15:21.035374 30696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:59133
I20260811 11:15:21.036082 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:15:21.041951 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.035310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:53573 (local address 127.28.25.62:33887)
0811 11:15:21.035365 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.035368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.035776 (+ 408us) server_negotiation.cc:407] Connection header received
0811 11:15:21.035911 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.035912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.035946 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.035983 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:21.036455 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.036944 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.037672 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.037889 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.037978 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.039122 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:21.039124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:21.039124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:21.039132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:21.040667 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:21.041104 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:21.041105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:21.041105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:21.041169 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:21.041388 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:21.041389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:21.041389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:21.041510 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:21.041589 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.041757 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.041850 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 11:15:21.041985 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.035407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33887 (local address 127.28.25.3:53573)
0811 11:15:21.035706 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.035719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.035822 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.036049 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.036050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.036277 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:21.036344 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.036347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.037061 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.037063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.037578 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.037579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.037983 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.037984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.038079 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.038629 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:21.038637 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:21.039024 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:21.040830 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:21.040832 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:21.040833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:21.041023 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:21.041239 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:21.041240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:21.041241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:21.041307 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:21.041580 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:21.041581 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:21.041637 (+ 56us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.041730 (+ 93us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.041880 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
Aug 11 11:15:21 dist-test-slave-2w3w krb5kdc[30645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 11:15:21 dist-test-slave-2w3w krb5kdc[30645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:21.068212 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.060321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.28.25.62:33887)
0811 11:15:21.060402 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.060404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.060413 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:21.060450 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.060450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.060460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.060523 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:21.061194 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.061710 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.062498 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.062706 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.062791 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.065363 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:21.065365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:21.065365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:21.065374 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:21.066827 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:21.067343 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:21.067344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:21.067344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:21.067409 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:21.067620 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:21.067621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:21.067621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:21.067748 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:21.067867 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.068000 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.068101 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:15:21.070380 30696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:15:21.072242 30696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:21.080040 31264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.075349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.28.25.3:44019)
0811 11:15:21.075960 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.075964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.075974 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:15:21.076075 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.076077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.076095 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.076170 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:21.076377 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.077551 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.078860 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.079504 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.079597 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.079667 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.079869 (+ 202us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.079919 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":111,"threads_started":1}
I20260811 11:15:21.080077 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.075334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.28.25.1:32859)
0811 11:15:21.075676 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.075682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.075698 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:15:21.075901 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.075905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.075926 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.075995 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:21.076380 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.077573 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.078819 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.079566 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.079612 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.079670 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.079920 (+ 250us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.079965 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":95,"threads_started":1}
I20260811 11:15:21.080056 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.075175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:32859 (local address 127.0.0.1:46134)
0811 11:15:21.075568 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.075620 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.075694 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.076197 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.076198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.076200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:21.076269 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.076271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.077663 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.077665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.078689 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.078692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.079732 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.079732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.079791 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.079795 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.079896 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.079964 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":31,"thread_start_us":74,"threads_started":1}
I20260811 11:15:21.080056 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.075414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:34293 (local address 127.0.0.1:32878)
0811 11:15:21.075851 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.075860 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.075920 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.076150 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.076153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.076156 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:21.076241 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.076243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.077303 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.077304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.078587 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.078590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.079714 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.079716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.079796 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.079799 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.079845 (+ 46us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.079949 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260811 11:15:21.080376 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.075544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.28.25.2:34293)
0811 11:15:21.075752 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.075756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.075991 (+ 235us) server_negotiation.cc:407] Connection header received
0811 11:15:21.076037 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.076039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.076055 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.076122 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:21.076337 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.077208 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.078690 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.079546 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.079624 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.079677 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.080100 (+ 423us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.080149 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":96,"threads_started":1}
I20260811 11:15:21.081130 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.075220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:44019 (local address 127.0.0.1:49932)
0811 11:15:21.075755 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.075765 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.075824 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.076241 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.076241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.076243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:21.076290 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.076291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.077647 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.077648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.078775 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.078777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.079611 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.079612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.079715 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.079731 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.079846 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.081052 (+ 1206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":1160,"thread_start_us":110,"threads_started":1}
I20260811 11:15:21.081461 31001 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb54a21d6c84b6597cc137ae99c9187 (DEFAULT_TABLE table=test-table [id=f44399fb330e416b9e108499f6ba8ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:21.081539 30852 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb54a21d6c84b6597cc137ae99c9187 (DEFAULT_TABLE table=test-table [id=f44399fb330e416b9e108499f6ba8ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:21.081766 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb54a21d6c84b6597cc137ae99c9187 (DEFAULT_TABLE table=test-table [id=f44399fb330e416b9e108499f6ba8ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:21.081892 31001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb54a21d6c84b6597cc137ae99c9187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:21.081892 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb54a21d6c84b6597cc137ae99c9187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:21.082055 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb54a21d6c84b6597cc137ae99c9187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:21.084823 31267 tablet_bootstrap.cc:492] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: Bootstrap starting.
I20260811 11:15:21.085909 31267 tablet_bootstrap.cc:654] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:21.085877 31269 tablet_bootstrap.cc:492] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: Bootstrap starting.
I20260811 11:15:21.086280 31267 log.cc:824] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:21.086615 31269 tablet_bootstrap.cc:654] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:21.087014 31269 log.cc:824] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:21.087174 31268 tablet_bootstrap.cc:492] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: Bootstrap starting.
I20260811 11:15:21.087378 31267 tablet_bootstrap.cc:492] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: No bootstrap required, opened a new log
I20260811 11:15:21.087471 31267 ts_tablet_manager.cc:1397] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:15:21.087904 31268 tablet_bootstrap.cc:654] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:21.087817 31269 tablet_bootstrap.cc:492] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: No bootstrap required, opened a new log
I20260811 11:15:21.087994 31269 ts_tablet_manager.cc:1397] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:21.088272 31268 log.cc:824] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:21.089593 31268 tablet_bootstrap.cc:492] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: No bootstrap required, opened a new log
I20260811 11:15:21.089679 31268 ts_tablet_manager.cc:1397] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 11:15:21.089540 31267 raft_consensus.cc:348] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.089720 31267 raft_consensus.cc:374] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:21.089776 31267 raft_consensus.cc:729] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040217d356e4373b2fa651d958dcc6b, State: Initialized, Role: FOLLOWER
I20260811 11:15:21.089901 31267 consensus_queue.cc:260] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.090090 31269 raft_consensus.cc:348] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.090232 31267 ts_tablet_manager.cc:1428] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:15:21.090263 31242 heartbeater.cc:503] Master 127.28.25.62:33887 was elected leader, sending a full tablet report...
I20260811 11:15:21.090260 31269 raft_consensus.cc:374] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:21.090306 31269 raft_consensus.cc:729] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26abc7b9c0734f31b2bb916adc9aea1d, State: Initialized, Role: FOLLOWER
I20260811 11:15:21.090420 31269 consensus_queue.cc:260] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.090682 31269 ts_tablet_manager.cc:1428] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:15:21.090845 30926 heartbeater.cc:503] Master 127.28.25.62:33887 was elected leader, sending a full tablet report...
I20260811 11:15:21.092041 31268 raft_consensus.cc:348] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.092180 31268 raft_consensus.cc:374] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:21.092201 31268 raft_consensus.cc:729] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7bb18d26df4a76bb342bc5d330d60d, State: Initialized, Role: FOLLOWER
I20260811 11:15:21.092315 31268 consensus_queue.cc:260] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.092563 31268 ts_tablet_manager.cc:1428] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:15:21.092650 31083 heartbeater.cc:503] Master 127.28.25.62:33887 was elected leader, sending a full tablet report...
W20260811 11:15:21.097797 31084 tablet.cc:2369] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:15:21.169431 30927 tablet.cc:2369] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:15:21.256129 31277 raft_consensus.cc:482] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:21.256270 31277 raft_consensus.cc:504] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.256803 31277 leader_election.cc:288] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859), 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019)
I20260811 11:15:21.259531 31276 raft_consensus.cc:482] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:21.259690 31276 raft_consensus.cc:504] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.260155 31276 leader_election.cc:288] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293), 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019)
I20260811 11:15:21.260347 31264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.257141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:48537 (local address 127.28.25.3:44019)
0811 11:15:21.257189 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.257191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.257332 (+ 141us) server_negotiation.cc:407] Connection header received
0811 11:15:21.257419 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.257420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.257437 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.257500 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:21.257669 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.258144 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.259344 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.259827 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.259915 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.259930 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.260209 (+ 279us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.260264 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:15:21.260416 30964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.257028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:32859 (local address 127.28.25.2:53345)
0811 11:15:21.257099 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.257114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.257162 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.257357 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.257358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.257360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:21.257436 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.257438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.258134 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.258135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.259231 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.259234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.260120 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.260121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.260216 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.260220 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.260274 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.260330 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":120}
I20260811 11:15:21.260653 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.257130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:53345 (local address 127.28.25.1:32859)
0811 11:15:21.257201 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.257203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.257223 (+ 20us) server_negotiation.cc:407] Connection header received
0811 11:15:21.257259 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.257260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.257271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.257338 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:21.257509 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.258022 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.259355 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.259973 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.260053 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.260064 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.260530 (+ 466us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.260587 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:15:21.261065 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.257055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:44019 (local address 127.28.25.2:48537)
0811 11:15:21.257257 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.257266 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.257347 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.257504 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.257505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.257506 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:21.257587 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.257589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.258244 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.258246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.259225 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.259230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.260028 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.260029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.260111 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.260116 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.260861 (+ 745us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.260913 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260811 11:15:21.261025 30874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb54a21d6c84b6597cc137ae99c9187" candidate_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" is_pre_election: true
I20260811 11:15:21.261137 30874 raft_consensus.cc:2454] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd7bb18d26df4a76bb342bc5d330d60d in term 0.
I20260811 11:15:21.261394 30956 leader_election.cc:302] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26abc7b9c0734f31b2bb916adc9aea1d, fd7bb18d26df4a76bb342bc5d330d60d; no voters:
I20260811 11:15:21.261544 31277 raft_consensus.cc:2781] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:21.261593 31277 raft_consensus.cc:482] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:21.261615 31277 raft_consensus.cc:3037] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:21.261727 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb54a21d6c84b6597cc137ae99c9187" candidate_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040217d356e4373b2fa651d958dcc6b" is_pre_election: true
I20260811 11:15:21.261879 31192 raft_consensus.cc:2454] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd7bb18d26df4a76bb342bc5d330d60d in term 0.
I20260811 11:15:21.262991 31277 raft_consensus.cc:504] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.263230 31277 leader_election.cc:288] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [CANDIDATE]: Term 1 election: Requested vote from peers 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859), 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019)
I20260811 11:15:21.263458 30874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb54a21d6c84b6597cc137ae99c9187" candidate_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26abc7b9c0734f31b2bb916adc9aea1d"
I20260811 11:15:21.263549 30874 raft_consensus.cc:3037] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:21.263697 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb54a21d6c84b6597cc137ae99c9187" candidate_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040217d356e4373b2fa651d958dcc6b"
I20260811 11:15:21.263804 31192 raft_consensus.cc:3037] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:21.263928 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.260434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:51201 (local address 127.28.25.2:34293)
0811 11:15:21.260525 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.260527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.260648 (+ 121us) server_negotiation.cc:407] Connection header received
0811 11:15:21.260738 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.260738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.260749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.260781 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:21.260999 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.261651 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.262810 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.263428 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.263513 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.263524 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.263798 (+ 274us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.263845 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 11:15:21.264279 31280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.260398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:34293 (local address 127.28.25.1:51201)
0811 11:15:21.260590 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.260601 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.260673 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.260847 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.260849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.260851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:21.260914 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.260916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.261747 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.261748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.262706 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.262709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.263527 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.263528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.263700 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.263706 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.264147 (+ 441us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.264205 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":107,"threads_started":1}
I20260811 11:15:21.264360 31264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.260791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:58607 (local address 127.28.25.3:44019)
0811 11:15:21.260830 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.260832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.261005 (+ 173us) server_negotiation.cc:407] Connection header received
0811 11:15:21.261113 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.261114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.261124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.261154 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:21.261379 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.262027 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.263394 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.263940 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.264028 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.264038 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.264240 (+ 202us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.264284 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 11:15:21.264533 30874 raft_consensus.cc:2454] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd7bb18d26df4a76bb342bc5d330d60d in term 1.
I20260811 11:15:21.264643 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.260635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:44019 (local address 127.28.25.1:58607)
0811 11:15:21.260871 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.260882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.261035 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.261221 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.261221 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.261223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:21.261291 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.261293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.262141 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.262142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.263256 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.263259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.264072 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.264073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.264149 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.264153 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.264514 (+ 361us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.264568 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":49,"threads_started":1}
I20260811 11:15:21.264853 31192 raft_consensus.cc:2454] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd7bb18d26df4a76bb342bc5d330d60d in term 1.
I20260811 11:15:21.264868 30956 leader_election.cc:302] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26abc7b9c0734f31b2bb916adc9aea1d, fd7bb18d26df4a76bb342bc5d330d60d; no voters:
I20260811 11:15:21.265054 31277 raft_consensus.cc:2781] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:21.265166 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb54a21d6c84b6597cc137ae99c9187" candidate_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" is_pre_election: true
I20260811 11:15:21.265290 31277 raft_consensus.cc:686] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 LEADER]: Becoming Leader. State: Replica: fd7bb18d26df4a76bb342bc5d330d60d, State: Running, Role: LEADER
I20260811 11:15:21.265416 31277 consensus_queue.cc:237] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.265415 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb54a21d6c84b6597cc137ae99c9187" candidate_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040217d356e4373b2fa651d958dcc6b" is_pre_election: true
I20260811 11:15:21.265539 31192 raft_consensus.cc:2379] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26abc7b9c0734f31b2bb916adc9aea1d in current term 1: Already voted for candidate fd7bb18d26df4a76bb342bc5d330d60d in this term.
I20260811 11:15:21.266307 30795 leader_election.cc:302] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26abc7b9c0734f31b2bb916adc9aea1d; no voters: 9040217d356e4373b2fa651d958dcc6b, fd7bb18d26df4a76bb342bc5d330d60d
I20260811 11:15:21.266450 31276 raft_consensus.cc:2726] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 11:15:21.266597 30695 catalog_manager.cc:5495] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d reported cstate change: term changed from 0 to 1, leader changed from <none> to fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2). New cstate: current_term: 1 leader_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } health_report { overall_health: UNKNOWN } } }
W20260811 11:15:21.280483 31243 tablet.cc:2369] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 11:15:21 dist-test-slave-2w3w krb5kdc[30645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446920, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:21.331516 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.323934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.28.25.62:33887)
0811 11:15:21.324052 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.324055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.324179 (+ 124us) server_negotiation.cc:407] Connection header received
0811 11:15:21.324306 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.324306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.324318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.324397 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:21.324911 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.325405 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.326119 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.326310 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.326400 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.328753 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:21.328755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:21.328756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:21.328763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:21.330171 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:21.330599 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:21.330600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:21.330601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:21.330662 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:21.330871 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:21.330872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:21.330873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:21.331060 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:21.331140 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.331303 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.331399 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 11:15:21.331596 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.323803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33887 (local address 127.0.0.1:42398)
0811 11:15:21.324090 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.324108 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.324202 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.324401 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.324402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.324748 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:21.324809 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.324813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.325489 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.325490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.326025 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.326026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.326401 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.326402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.326457 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.327113 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:21.327119 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:21.328644 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:21.330267 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:21.330279 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:21.330282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:21.330517 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:21.330735 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:21.330735 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:21.330736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:21.330803 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:21.331129 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:21.331139 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:21.331190 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.331279 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.331459 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":57,"thread_start_us":170,"threads_started":1}
I20260811 11:15:21.334072 30695 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:42398:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 11:15:21.334297 30695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:21.336124 31001 tablet_service.cc:1467] Processing CreateTablet for tablet da91ece95f054d9e93f0200051be2e32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd7ae7eefd9d4caaac2dae7aaaa61536]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:15:21.336144 31169 tablet_service.cc:1467] Processing CreateTablet for tablet da91ece95f054d9e93f0200051be2e32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd7ae7eefd9d4caaac2dae7aaaa61536]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:15:21.336349 31001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da91ece95f054d9e93f0200051be2e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:21.336355 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da91ece95f054d9e93f0200051be2e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:21.337808 31268 tablet_bootstrap.cc:492] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: Bootstrap starting.
I20260811 11:15:21.337939 31267 tablet_bootstrap.cc:492] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: Bootstrap starting.
I20260811 11:15:21.338019 30852 tablet_service.cc:1467] Processing CreateTablet for tablet da91ece95f054d9e93f0200051be2e32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd7ae7eefd9d4caaac2dae7aaaa61536]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:15:21.338213 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da91ece95f054d9e93f0200051be2e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:21.338420 31268 tablet_bootstrap.cc:654] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:21.338560 31267 tablet_bootstrap.cc:654] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:21.339309 31268 tablet_bootstrap.cc:492] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: No bootstrap required, opened a new log
I20260811 11:15:21.339370 31268 ts_tablet_manager.cc:1397] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:21.339553 31268 raft_consensus.cc:348] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.339644 31268 raft_consensus.cc:374] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:21.339672 31268 raft_consensus.cc:729] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7bb18d26df4a76bb342bc5d330d60d, State: Initialized, Role: FOLLOWER
I20260811 11:15:21.339721 31268 consensus_queue.cc:260] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.339828 31268 ts_tablet_manager.cc:1428] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 11:15:21.339895 31277 tablet_replica.cc:439] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } } }
I20260811 11:15:21.340019 31277 tablet_replica.cc:442] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:21.340034 31269 tablet_bootstrap.cc:492] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: Bootstrap starting.
I20260811 11:15:21.340479 31267 tablet_bootstrap.cc:492] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: No bootstrap required, opened a new log
I20260811 11:15:21.340528 31267 ts_tablet_manager.cc:1397] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260811 11:15:21.340546 31269 tablet_bootstrap.cc:654] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:21.340700 31267 raft_consensus.cc:348] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.340757 31267 raft_consensus.cc:374] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:21.340771 31267 raft_consensus.cc:729] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040217d356e4373b2fa651d958dcc6b, State: Initialized, Role: FOLLOWER
I20260811 11:15:21.340838 31267 consensus_queue.cc:260] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.340945 31267 ts_tablet_manager.cc:1428] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:21.341061 31275 tablet_replica.cc:439] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } } }
I20260811 11:15:21.341172 31275 tablet_replica.cc:442] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:21.341634 31269 tablet_bootstrap.cc:492] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: No bootstrap required, opened a new log
I20260811 11:15:21.341692 31269 ts_tablet_manager.cc:1397] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:21.341856 31269 raft_consensus.cc:348] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.341925 31269 raft_consensus.cc:374] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:21.341945 31269 raft_consensus.cc:729] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26abc7b9c0734f31b2bb916adc9aea1d, State: Initialized, Role: FOLLOWER
I20260811 11:15:21.342015 31269 consensus_queue.cc:260] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.342109 31269 ts_tablet_manager.cc:1428] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:21.342170 31276 tablet_replica.cc:439] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } } }
I20260811 11:15:21.342286 31276 tablet_replica.cc:442] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:21.394594 31276 raft_consensus.cc:482] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:21.394730 31276 raft_consensus.cc:504] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.395013 31276 leader_election.cc:288] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293), 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019)
I20260811 11:15:21.395238 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da91ece95f054d9e93f0200051be2e32" candidate_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" is_pre_election: true
I20260811 11:15:21.395404 31033 raft_consensus.cc:2454] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26abc7b9c0734f31b2bb916adc9aea1d in term 0.
I20260811 11:15:21.395342 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da91ece95f054d9e93f0200051be2e32" candidate_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040217d356e4373b2fa651d958dcc6b" is_pre_election: true
I20260811 11:15:21.395483 31192 raft_consensus.cc:2454] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26abc7b9c0734f31b2bb916adc9aea1d in term 0.
I20260811 11:15:21.395669 30795 leader_election.cc:302] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26abc7b9c0734f31b2bb916adc9aea1d, fd7bb18d26df4a76bb342bc5d330d60d; no voters:
I20260811 11:15:21.395807 31276 raft_consensus.cc:2781] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:21.395854 31276 raft_consensus.cc:482] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:21.395881 31276 raft_consensus.cc:3037] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:21.396692 31276 raft_consensus.cc:504] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.396857 31276 leader_election.cc:288] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [CANDIDATE]: Term 1 election: Requested vote from peers fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293), 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019)
I20260811 11:15:21.397008 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da91ece95f054d9e93f0200051be2e32" candidate_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7bb18d26df4a76bb342bc5d330d60d"
I20260811 11:15:21.397126 31033 raft_consensus.cc:3037] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:21.397080 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da91ece95f054d9e93f0200051be2e32" candidate_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040217d356e4373b2fa651d958dcc6b"
I20260811 11:15:21.397197 31192 raft_consensus.cc:3037] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:21.397914 31192 raft_consensus.cc:2454] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26abc7b9c0734f31b2bb916adc9aea1d in term 1.
I20260811 11:15:21.397922 31033 raft_consensus.cc:2454] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26abc7b9c0734f31b2bb916adc9aea1d in term 1.
I20260811 11:15:21.398113 30795 leader_election.cc:302] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26abc7b9c0734f31b2bb916adc9aea1d, fd7bb18d26df4a76bb342bc5d330d60d; no voters:
I20260811 11:15:21.398244 31276 raft_consensus.cc:2781] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:21.398368 31276 raft_consensus.cc:686] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 LEADER]: Becoming Leader. State: Replica: 26abc7b9c0734f31b2bb916adc9aea1d, State: Running, Role: LEADER
I20260811 11:15:21.398485 31276 consensus_queue.cc:237] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.399015 31294 tablet_replica.cc:439] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: TxnStatusTablet state changed. Reason: New leader 26abc7b9c0734f31b2bb916adc9aea1d. Latest consensus state: current_term: 1 leader_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } } }
I20260811 11:15:21.399121 31294 tablet_replica.cc:442] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:15:21.399348 31296 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 11:15:21.399501 31296 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 11:15:21.399510 30695 catalog_manager.cc:5495] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1). New cstate: current_term: 1 leader_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } health_report { overall_health: UNKNOWN } } }
I20260811 11:15:21.416409 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.413486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:32859 (local address 127.0.0.1:46138)
0811 11:15:21.413573 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.413603 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.413670 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.413948 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.413949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.413951 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:15:21.414009 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.414017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.414726 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.414727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.415443 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.415445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.415833 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.415835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.415906 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.415925 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:15:21.416227 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:15:21.416240 (+ 13us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.416282 (+ 42us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.416339 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260811 11:15:21.416400 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.413560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.28.25.1:32859)
0811 11:15:21.413632 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.413634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.413650 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:15:21.413832 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.413833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.413856 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.413943 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:21.414097 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.414623 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.415536 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.415745 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.415820 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.416033 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:21.416153 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:21.416262 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.416285 (+ 23us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.416311 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:15:21.418517 31033 raft_consensus.cc:1261] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Refusing update from remote peer 26abc7b9c0734f31b2bb916adc9aea1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:21.418524 31192 raft_consensus.cc:1261] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Refusing update from remote peer 26abc7b9c0734f31b2bb916adc9aea1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:21.418887 31276 consensus_queue.cc:1035] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:21.419075 31294 consensus_queue.cc:1035] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:21.420499 31275 tablet_replica.cc:439] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: TxnStatusTablet state changed. Reason: New leader 26abc7b9c0734f31b2bb916adc9aea1d. Latest consensus state: current_term: 1 leader_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } } }
I20260811 11:15:21.420615 31275 tablet_replica.cc:442] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:21.420697 31277 tablet_replica.cc:439] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: TxnStatusTablet state changed. Reason: New leader 26abc7b9c0734f31b2bb916adc9aea1d. Latest consensus state: current_term: 1 leader_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } } }
I20260811 11:15:21.420812 31277 tablet_replica.cc:442] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:21.421859 31294 tablet_replica.cc:439] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } } }
I20260811 11:15:21.421963 31294 tablet_replica.cc:442] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:15:21.422014 31294 tablet_replica.cc:439] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } } }
I20260811 11:15:21.422052 31294 tablet_replica.cc:442] T da91ece95f054d9e93f0200051be2e32 P 26abc7b9c0734f31b2bb916adc9aea1d: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:15:21.422050 31275 tablet_replica.cc:439] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } } }
I20260811 11:15:21.422137 31275 tablet_replica.cc:442] T da91ece95f054d9e93f0200051be2e32 P 9040217d356e4373b2fa651d958dcc6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:21.422773 31277 tablet_replica.cc:439] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } } }
I20260811 11:15:21.422878 31277 tablet_replica.cc:442] T da91ece95f054d9e93f0200051be2e32 P fd7bb18d26df4a76bb342bc5d330d60d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:15:21.423305 31303 mvcc.cc:205] Tried to move back new op lower bound from 7317286590127022080 to 7317286590048436224. Current Snapshot: MvccSnapshot[applied={T|T < 7317286590127022080}]
I20260811 11:15:21.424226 31302 mvcc.cc:205] Tried to move back new op lower bound from 7317286590127022080 to 7317286590048436224. Current Snapshot: MvccSnapshot[applied={T|T < 7317286590127022080}]
I20260811 11:15:21.427291 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.424186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.28.25.2:34293)
0811 11:15:21.424229 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.424231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.424240 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:21.424277 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.424278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.424300 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.424384 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:21.424619 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.425176 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.426393 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.426596 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.426661 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.426857 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:21.427044 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:21.427134 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.427199 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.427222 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 11:15:21.433364 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.430191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:33133 (local address 127.28.25.1:32859)
0811 11:15:21.430264 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.430267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.430497 (+ 230us) server_negotiation.cc:407] Connection header received
0811 11:15:21.430565 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.430565 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.430576 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.430626 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:21.430811 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.431383 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.432558 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.432770 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.432841 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.433097 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:21.433130 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:21.433172 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.433260 (+ 88us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.433282 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:15:21.433418 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.430116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:32859 (local address 127.28.25.2:33133)
0811 11:15:21.430353 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.430366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.430500 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.430639 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.430640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.430642 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:15:21.430713 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.430715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.431465 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.431466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.432456 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.432461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.432865 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.432866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.432944 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.432973 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:15:21.433195 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:15:21.433197 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.433247 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.433300 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":142,"threads_started":1}
I20260811 11:15:21.436530 31192 raft_consensus.cc:1261] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Refusing update from remote peer fd7bb18d26df4a76bb342bc5d330d60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:21.436661 30874 raft_consensus.cc:1261] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Refusing update from remote peer fd7bb18d26df4a76bb342bc5d330d60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:21.436815 31277 consensus_queue.cc:1035] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:21.437043 31277 consensus_queue.cc:1035] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:21.439007 31301 mvcc.cc:205] Tried to move back new op lower bound from 7317286590201970688 to 7317286589503897600. Current Snapshot: MvccSnapshot[applied={T|T < 7317286590201970688}]
I20260811 11:15:21.454421 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.445687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:60279 (local address 127.28.25.2:34293)
0811 11:15:21.445800 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.445802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.451187 (+ 5385us) server_negotiation.cc:407] Connection header received
0811 11:15:21.451226 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.451227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.451239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.451313 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:21.451525 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.452117 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.453358 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.453713 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.453791 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.454082 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:21.454142 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:21.454211 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.454302 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.454326 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 11:15:21.454911 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.445631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:34293 (local address 127.28.25.1:60279)
0811 11:15:21.451098 (+ 5467us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.451111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.451182 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.451346 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.451347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.451348 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:15:21.451431 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.451433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.452207 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.452208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.453194 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.453198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.453840 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.453842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.453931 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.453961 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:15:21.454227 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:15:21.454230 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.454295 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.454352 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5387,"thread_start_us":193,"threads_started":1}
I20260811 11:15:21.469296 30696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:21.469444 30696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:21.470353 30696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927: Sending DeleteTablet for 3 replicas of tablet 1cb54a21d6c84b6597cc137ae99c9187
I20260811 11:15:21.471010 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb54a21d6c84b6597cc137ae99c9187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:21 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:49932
I20260811 11:15:21.471029 31001 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb54a21d6c84b6597cc137ae99c9187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:21 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:32878
I20260811 11:15:21.471397 30852 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb54a21d6c84b6597cc137ae99c9187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:21 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:46134
I20260811 11:15:21.471678 31325 tablet_replica.cc:330] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: stopping tablet replica
I20260811 11:15:21.471814 31325 raft_consensus.cc:2229] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:15:21.471740 30696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:15:21.471900 30696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:21.471935 31325 raft_consensus.cc:2258] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:21.472134 31324 tablet_replica.cc:330] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: stopping tablet replica
I20260811 11:15:21.472267 31324 raft_consensus.cc:2229] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:21.472340 31324 raft_consensus.cc:2258] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:21.472930 31325 ts_tablet_manager.cc:1905] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:21.473500 30852 tablet_service.cc:1467] Processing CreateTablet for tablet 9324e27011cd4b819503e0af63905595 (DEFAULT_TABLE table=test-table [id=f9ef74ec8fbe44e89870632ec164e5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:21.473639 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9324e27011cd4b819503e0af63905595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:21.475049 31001 tablet_service.cc:1467] Processing CreateTablet for tablet 9324e27011cd4b819503e0af63905595 (DEFAULT_TABLE table=test-table [id=f9ef74ec8fbe44e89870632ec164e5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:21.475306 31001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9324e27011cd4b819503e0af63905595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:21.475298 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 9324e27011cd4b819503e0af63905595 (DEFAULT_TABLE table=test-table [id=f9ef74ec8fbe44e89870632ec164e5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:21.475493 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9324e27011cd4b819503e0af63905595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:21.476819 31268 tablet_bootstrap.cc:492] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d: Bootstrap starting.
I20260811 11:15:21.476979 31267 tablet_bootstrap.cc:492] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b: Bootstrap starting.
I20260811 11:15:21.477051 31326 tablet_replica.cc:330] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: stopping tablet replica
I20260811 11:15:21.477223 31326 raft_consensus.cc:2229] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:21.477300 31326 raft_consensus.cc:2258] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:21.477394 31268 tablet_bootstrap.cc:654] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:21.477427 31267 tablet_bootstrap.cc:654] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:21.477785 31325 ts_tablet_manager.cc:1918] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:15:21.477873 31325 log.cc:1196] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/1cb54a21d6c84b6597cc137ae99c9187
I20260811 11:15:21.477929 31326 ts_tablet_manager.cc:1905] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:21.478193 31325 ts_tablet_manager.cc:1939] T 1cb54a21d6c84b6597cc137ae99c9187 P fd7bb18d26df4a76bb342bc5d330d60d: Deleting consensus metadata
I20260811 11:15:21.478672 30679 catalog_manager.cc:4841] TS fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293): tablet 1cb54a21d6c84b6597cc137ae99c9187 (table test-table [id=f44399fb330e416b9e108499f6ba8ffe]) successfully deleted
I20260811 11:15:21.479566 31269 tablet_bootstrap.cc:492] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d: Bootstrap starting.
I20260811 11:15:21.479782 31326 ts_tablet_manager.cc:1918] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:15:21.479871 31326 log.cc:1196] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/1cb54a21d6c84b6597cc137ae99c9187
I20260811 11:15:21.480005 31269 tablet_bootstrap.cc:654] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:21.480175 31326 ts_tablet_manager.cc:1939] T 1cb54a21d6c84b6597cc137ae99c9187 P 26abc7b9c0734f31b2bb916adc9aea1d: Deleting consensus metadata
I20260811 11:15:21.481035 31269 tablet_bootstrap.cc:492] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d: No bootstrap required, opened a new log
I20260811 11:15:21.481096 31269 ts_tablet_manager.cc:1397] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:21.481205 31324 ts_tablet_manager.cc:1905] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:21.481266 31269 raft_consensus.cc:348] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.481364 31269 raft_consensus.cc:374] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:21.481391 31269 raft_consensus.cc:729] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26abc7b9c0734f31b2bb916adc9aea1d, State: Initialized, Role: FOLLOWER
I20260811 11:15:21.481436 31269 consensus_queue.cc:260] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.481546 31269 ts_tablet_manager.cc:1428] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 11:15:21.483716 31324 ts_tablet_manager.cc:1918] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:15:21.483824 31324 log.cc:1196] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/1cb54a21d6c84b6597cc137ae99c9187
I20260811 11:15:21.484097 30679 catalog_manager.cc:4841] TS 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859): tablet 1cb54a21d6c84b6597cc137ae99c9187 (table test-table [id=f44399fb330e416b9e108499f6ba8ffe]) successfully deleted
I20260811 11:15:21.484215 31324 ts_tablet_manager.cc:1939] T 1cb54a21d6c84b6597cc137ae99c9187 P 9040217d356e4373b2fa651d958dcc6b: Deleting consensus metadata
I20260811 11:15:21.484656 31267 tablet_bootstrap.cc:492] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b: No bootstrap required, opened a new log
I20260811 11:15:21.484694 30680 catalog_manager.cc:4841] TS 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019): tablet 1cb54a21d6c84b6597cc137ae99c9187 (table test-table [id=f44399fb330e416b9e108499f6ba8ffe]) successfully deleted
I20260811 11:15:21.484714 31267 ts_tablet_manager.cc:1397] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260811 11:15:21.484889 31267 raft_consensus.cc:348] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.484980 31267 raft_consensus.cc:374] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:21.485004 31267 raft_consensus.cc:729] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040217d356e4373b2fa651d958dcc6b, State: Initialized, Role: FOLLOWER
I20260811 11:15:21.485054 31267 consensus_queue.cc:260] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.485153 31267 ts_tablet_manager.cc:1428] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:21.485826 31268 tablet_bootstrap.cc:492] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d: No bootstrap required, opened a new log
I20260811 11:15:21.485905 31268 ts_tablet_manager.cc:1397] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260811 11:15:21.486092 31268 raft_consensus.cc:348] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.486171 31268 raft_consensus.cc:374] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:21.486186 31268 raft_consensus.cc:729] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7bb18d26df4a76bb342bc5d330d60d, State: Initialized, Role: FOLLOWER
I20260811 11:15:21.486254 31268 consensus_queue.cc:260] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.486370 31268 ts_tablet_manager.cc:1428] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:21.720882 31314 raft_consensus.cc:482] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:21.721024 31314 raft_consensus.cc:504] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.721319 31314 leader_election.cc:288] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293), 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019)
I20260811 11:15:21.721627 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9324e27011cd4b819503e0af63905595" candidate_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040217d356e4373b2fa651d958dcc6b" is_pre_election: true
I20260811 11:15:21.721648 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9324e27011cd4b819503e0af63905595" candidate_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" is_pre_election: true
I20260811 11:15:21.721752 31033 raft_consensus.cc:2454] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26abc7b9c0734f31b2bb916adc9aea1d in term 0.
I20260811 11:15:21.721746 31192 raft_consensus.cc:2454] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26abc7b9c0734f31b2bb916adc9aea1d in term 0.
I20260811 11:15:21.721973 30795 leader_election.cc:302] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26abc7b9c0734f31b2bb916adc9aea1d, fd7bb18d26df4a76bb342bc5d330d60d; no voters:
I20260811 11:15:21.722095 31314 raft_consensus.cc:2781] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:21.722184 31314 raft_consensus.cc:482] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:21.722222 31314 raft_consensus.cc:3037] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:21.723337 31314 raft_consensus.cc:504] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.723531 31314 leader_election.cc:288] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [CANDIDATE]: Term 1 election: Requested vote from peers fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293), 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019)
I20260811 11:15:21.723742 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9324e27011cd4b819503e0af63905595" candidate_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7bb18d26df4a76bb342bc5d330d60d"
I20260811 11:15:21.723845 31033 raft_consensus.cc:3037] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:21.723830 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9324e27011cd4b819503e0af63905595" candidate_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040217d356e4373b2fa651d958dcc6b"
I20260811 11:15:21.723973 31192 raft_consensus.cc:3037] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:21.724653 31033 raft_consensus.cc:2454] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26abc7b9c0734f31b2bb916adc9aea1d in term 1.
I20260811 11:15:21.724681 31192 raft_consensus.cc:2454] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26abc7b9c0734f31b2bb916adc9aea1d in term 1.
I20260811 11:15:21.724920 30795 leader_election.cc:302] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26abc7b9c0734f31b2bb916adc9aea1d, fd7bb18d26df4a76bb342bc5d330d60d; no voters:
I20260811 11:15:21.725050 31314 raft_consensus.cc:2781] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:21.725136 31314 raft_consensus.cc:686] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 LEADER]: Becoming Leader. State: Replica: 26abc7b9c0734f31b2bb916adc9aea1d, State: Running, Role: LEADER
I20260811 11:15:21.725199 31314 consensus_queue.cc:237] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:21.725960 30696 catalog_manager.cc:5495] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1). New cstate: current_term: 1 leader_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } health_report { overall_health: UNKNOWN } } }
I20260811 11:15:21.769505 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.767247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.28.25.1:32859)
0811 11:15:21.767299 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.767301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.767310 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:21.767335 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.767335 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.767345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.767390 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:21.767520 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.768013 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.768773 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.768991 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.769069 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.769222 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:21.769258 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:21.769321 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.769391 (+ 70us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.769410 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:15:21.774504 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.772112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:60089 (local address 127.28.25.1:32859)
0811 11:15:21.772185 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:21.772187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:21.772196 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:21.772215 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:21.772216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:21.772224 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:21.772275 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:21.772392 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.772964 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.773841 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.774026 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.774090 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.774211 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:21.774236 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:21.774313 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:21.774362 (+ 49us) server_negotiation.cc:299] Negotiation successful
0811 11:15:21.774394 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:15:21.774636 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:21.771991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:32859 (local address 127.28.25.1:60089)
0811 11:15:21.772061 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:21.772074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:21.772124 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:21.772287 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:21.772287 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:21.772289 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:15:21.772328 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.772329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.773031 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.773031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.773734 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:21.773737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:21.774098 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:21.774099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:21.774142 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:21.774148 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:15:21.774320 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:15:21.774321 (+ 1us) client_negotiation.cc:769] Sending connection context
0811 11:15:21.774358 (+ 37us) client_negotiation.cc:240] Negotiation successful
0811 11:15:21.774426 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1}
I20260811 11:15:21.776896 31192 raft_consensus.cc:1261] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Refusing update from remote peer 26abc7b9c0734f31b2bb916adc9aea1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:21.777016 31032 raft_consensus.cc:1261] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Refusing update from remote peer 26abc7b9c0734f31b2bb916adc9aea1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:21.777218 31294 consensus_queue.cc:1035] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:21.777303 31276 consensus_queue.cc:1035] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 11:15:21.782320 31301 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 11:15:23.303887 30695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:23.304021 30695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:23.304999 30695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927: Sending DeleteTablet for 3 replicas of tablet 9324e27011cd4b819503e0af63905595
I20260811 11:15:23.305338 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 9324e27011cd4b819503e0af63905595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:23 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:49932
I20260811 11:15:23.305410 31001 tablet_service.cc:1514] Processing DeleteTablet for tablet 9324e27011cd4b819503e0af63905595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:23 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:32878
I20260811 11:15:23.305560 30851 tablet_service.cc:1514] Processing DeleteTablet for tablet 9324e27011cd4b819503e0af63905595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:23 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:46134
I20260811 11:15:23.305768 31358 tablet_replica.cc:330] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d: stopping tablet replica
I20260811 11:15:23.305856 31358 raft_consensus.cc:2229] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:15:23.305950 31358 raft_consensus.cc:2258] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:23.306061 31357 tablet_replica.cc:330] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d: stopping tablet replica
I20260811 11:15:23.306124 31357 raft_consensus.cc:2229] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:23.306169 31357 raft_consensus.cc:2258] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:23.309577 31356 tablet_replica.cc:330] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b: stopping tablet replica
I20260811 11:15:23.309612 31358 ts_tablet_manager.cc:1905] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:23.309772 31356 raft_consensus.cc:2229] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:23.309875 31356 raft_consensus.cc:2258] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:23.311286 31357 ts_tablet_manager.cc:1905] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:23.311512 31358 ts_tablet_manager.cc:1918] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 11:15:23.311589 31358 log.cc:1196] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/9324e27011cd4b819503e0af63905595
I20260811 11:15:23.311918 31358 ts_tablet_manager.cc:1939] T 9324e27011cd4b819503e0af63905595 P 26abc7b9c0734f31b2bb916adc9aea1d: Deleting consensus metadata
I20260811 11:15:23.312378 31356 ts_tablet_manager.cc:1905] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:23.313040 31357 ts_tablet_manager.cc:1918] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 11:15:23.313124 31357 log.cc:1196] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/9324e27011cd4b819503e0af63905595
I20260811 11:15:23.313370 31357 ts_tablet_manager.cc:1939] T 9324e27011cd4b819503e0af63905595 P fd7bb18d26df4a76bb342bc5d330d60d: Deleting consensus metadata
I20260811 11:15:23.314229 30679 catalog_manager.cc:4841] TS fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293): tablet 9324e27011cd4b819503e0af63905595 (table test-table [id=f9ef74ec8fbe44e89870632ec164e5ed]) successfully deleted
I20260811 11:15:23.314329 30679 catalog_manager.cc:4841] TS 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859): tablet 9324e27011cd4b819503e0af63905595 (table test-table [id=f9ef74ec8fbe44e89870632ec164e5ed]) successfully deleted
I20260811 11:15:23.314391 31356 ts_tablet_manager.cc:1918] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 11:15:23.314460 31356 log.cc:1196] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/9324e27011cd4b819503e0af63905595
I20260811 11:15:23.314690 31356 ts_tablet_manager.cc:1939] T 9324e27011cd4b819503e0af63905595 P 9040217d356e4373b2fa651d958dcc6b: Deleting consensus metadata
I20260811 11:15:23.315138 30680 catalog_manager.cc:4841] TS 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019): tablet 9324e27011cd4b819503e0af63905595 (table test-table [id=f9ef74ec8fbe44e89870632ec164e5ed]) successfully deleted
Aug 11 11:15:23 dist-test-slave-2w3w krb5kdc[30645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 11:15:23 dist-test-slave-2w3w krb5kdc[30645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:23.334800 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.325942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.28.25.62:33887)
0811 11:15:23.326226 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:23.326228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:23.326238 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:15:23.326266 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:23.326267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:23.326278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:23.326344 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:23.327432 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.327969 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.328730 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.328984 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.329052 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.331935 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:23.331937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:23.331937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:23.331946 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:23.333700 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:23.334064 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:23.334065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:23.334065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:23.334121 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:23.334273 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:23.334274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:23.334274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:23.334436 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:23.334560 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:23.334640 (+ 80us) server_negotiation.cc:299] Negotiation successful
0811 11:15:23.334723 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":223,"threads_started":1}
I20260811 11:15:23.336572 30695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:15:23.336764 30695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:23.338282 30851 tablet_service.cc:1467] Processing CreateTablet for tablet 1097b8ee1a6a491fab81345af4196483 (DEFAULT_TABLE table=test-table [id=9aaee3ef89f9490cb87bfefb2a30e533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:23.338330 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 1097b8ee1a6a491fab81345af4196483 (DEFAULT_TABLE table=test-table [id=9aaee3ef89f9490cb87bfefb2a30e533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:23.338330 31001 tablet_service.cc:1467] Processing CreateTablet for tablet 1097b8ee1a6a491fab81345af4196483 (DEFAULT_TABLE table=test-table [id=9aaee3ef89f9490cb87bfefb2a30e533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:23.338478 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1097b8ee1a6a491fab81345af4196483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:23.338491 31001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1097b8ee1a6a491fab81345af4196483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:23.338480 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1097b8ee1a6a491fab81345af4196483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:23.340179 31367 tablet_bootstrap.cc:492] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d: Bootstrap starting.
I20260811 11:15:23.340417 31369 tablet_bootstrap.cc:492] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b: Bootstrap starting.
I20260811 11:15:23.340581 31368 tablet_bootstrap.cc:492] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d: Bootstrap starting.
I20260811 11:15:23.340834 31367 tablet_bootstrap.cc:654] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:23.341109 31368 tablet_bootstrap.cc:654] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:23.341419 31369 tablet_bootstrap.cc:654] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:23.341775 31367 tablet_bootstrap.cc:492] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d: No bootstrap required, opened a new log
I20260811 11:15:23.341918 31367 ts_tablet_manager.cc:1397] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:23.342131 31367 raft_consensus.cc:348] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.342267 31367 raft_consensus.cc:374] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:23.342293 31367 raft_consensus.cc:729] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26abc7b9c0734f31b2bb916adc9aea1d, State: Initialized, Role: FOLLOWER
I20260811 11:15:23.342382 31367 consensus_queue.cc:260] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.342504 31367 ts_tablet_manager.cc:1428] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:23.342744 31369 tablet_bootstrap.cc:492] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b: No bootstrap required, opened a new log
I20260811 11:15:23.342806 31369 ts_tablet_manager.cc:1397] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:15:23.343014 31368 tablet_bootstrap.cc:492] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d: No bootstrap required, opened a new log
I20260811 11:15:23.343158 31368 ts_tablet_manager.cc:1397] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 11:15:23.343287 31369 raft_consensus.cc:348] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.343287 31368 raft_consensus.cc:348] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.343379 31369 raft_consensus.cc:374] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:23.343384 31368 raft_consensus.cc:374] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:23.343400 31369 raft_consensus.cc:729] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040217d356e4373b2fa651d958dcc6b, State: Initialized, Role: FOLLOWER
I20260811 11:15:23.343411 31368 raft_consensus.cc:729] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7bb18d26df4a76bb342bc5d330d60d, State: Initialized, Role: FOLLOWER
I20260811 11:15:23.343472 31369 consensus_queue.cc:260] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.343516 31368 consensus_queue.cc:260] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.343739 31369 ts_tablet_manager.cc:1428] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:23.343959 31368 ts_tablet_manager.cc:1428] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:23.380872 31374 raft_consensus.cc:482] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:23.381016 31374 raft_consensus.cc:504] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.381439 31374 leader_election.cc:288] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859), fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293)
I20260811 11:15:23.386263 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.382159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:35345 (local address 127.28.25.2:34293)
0811 11:15:23.382356 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:23.382358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:23.382365 (+ 7us) server_negotiation.cc:407] Connection header received
0811 11:15:23.382390 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:23.382391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:23.382401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:23.382456 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:23.382637 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.383427 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.385110 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.385720 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.385805 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.385819 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:23.386126 (+ 307us) server_negotiation.cc:299] Negotiation successful
0811 11:15:23.386178 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":115,"threads_started":1}
I20260811 11:15:23.386267 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.381770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:33679 (local address 127.28.25.1:32859)
0811 11:15:23.382008 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:23.382010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:23.382020 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:15:23.382068 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:23.382069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:23.382081 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:23.382205 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:23.382575 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.383398 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.385111 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.385720 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.385805 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.385819 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:23.386151 (+ 332us) server_negotiation.cc:299] Negotiation successful
0811 11:15:23.386200 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":125,"threads_started":1}
I20260811 11:15:23.386276 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.381683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:32859 (local address 127.28.25.3:33679)
0811 11:15:23.381905 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:23.381919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:23.381982 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:23.382225 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:23.382226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:23.382229 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:23.382468 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:23.382472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.383504 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.383505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:23.384992 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:23.384997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.385837 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.385837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:23.386071 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.386076 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:15:23.386129 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 11:15:23.386161 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":140,"thread_start_us":109,"threads_started":1}
I20260811 11:15:23.386281 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.381944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:34293 (local address 127.28.25.3:35345)
0811 11:15:23.382122 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:23.382135 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:23.382187 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:23.382492 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:23.382493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:23.382495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:23.382560 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:23.382561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.383504 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.383505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:23.384992 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:23.384997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.385835 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.385837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:23.386006 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.386013 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:15:23.386094 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 11:15:23.386156 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":9,"thread_start_us":97,"threads_started":1}
I20260811 11:15:23.386663 30874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1097b8ee1a6a491fab81345af4196483" candidate_uuid: "9040217d356e4373b2fa651d958dcc6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" is_pre_election: true
I20260811 11:15:23.386761 30874 raft_consensus.cc:2454] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9040217d356e4373b2fa651d958dcc6b in term 0.
I20260811 11:15:23.386773 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1097b8ee1a6a491fab81345af4196483" candidate_uuid: "9040217d356e4373b2fa651d958dcc6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" is_pre_election: true
I20260811 11:15:23.386884 31032 raft_consensus.cc:2454] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9040217d356e4373b2fa651d958dcc6b in term 0.
I20260811 11:15:23.387130 31111 leader_election.cc:302] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26abc7b9c0734f31b2bb916adc9aea1d, 9040217d356e4373b2fa651d958dcc6b; no voters:
I20260811 11:15:23.387291 31374 raft_consensus.cc:2781] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:23.387365 31374 raft_consensus.cc:482] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:23.387405 31374 raft_consensus.cc:3037] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:23.388746 31374 raft_consensus.cc:504] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.388916 31374 leader_election.cc:288] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [CANDIDATE]: Term 1 election: Requested vote from peers 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859), fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293)
I20260811 11:15:23.389202 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1097b8ee1a6a491fab81345af4196483" candidate_uuid: "9040217d356e4373b2fa651d958dcc6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7bb18d26df4a76bb342bc5d330d60d"
I20260811 11:15:23.389180 30874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1097b8ee1a6a491fab81345af4196483" candidate_uuid: "9040217d356e4373b2fa651d958dcc6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26abc7b9c0734f31b2bb916adc9aea1d"
I20260811 11:15:23.389302 31032 raft_consensus.cc:3037] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:23.389304 30874 raft_consensus.cc:3037] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:23.390363 30874 raft_consensus.cc:2454] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040217d356e4373b2fa651d958dcc6b in term 1.
I20260811 11:15:23.390363 31032 raft_consensus.cc:2454] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040217d356e4373b2fa651d958dcc6b in term 1.
I20260811 11:15:23.390602 31111 leader_election.cc:302] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26abc7b9c0734f31b2bb916adc9aea1d, 9040217d356e4373b2fa651d958dcc6b; no voters:
I20260811 11:15:23.390717 31374 raft_consensus.cc:2781] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:23.390925 31374 raft_consensus.cc:686] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 1 LEADER]: Becoming Leader. State: Replica: 9040217d356e4373b2fa651d958dcc6b, State: Running, Role: LEADER
I20260811 11:15:23.391042 31374 consensus_queue.cc:237] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.391858 30695 catalog_manager.cc:5495] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9040217d356e4373b2fa651d958dcc6b (127.28.25.3). New cstate: current_term: 1 leader_uuid: "9040217d356e4373b2fa651d958dcc6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } health_report { overall_health: HEALTHY } } }
I20260811 11:15:23.407603 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.404619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.28.25.3:44019)
0811 11:15:23.404831 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:23.404833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:23.404839 (+ 6us) server_negotiation.cc:407] Connection header received
0811 11:15:23.404883 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:23.404884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:23.404912 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:23.404982 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:23.405136 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.405678 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.406551 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.406761 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.406814 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.407109 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:23.407229 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:23.407291 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:23.407431 (+ 140us) server_negotiation.cc:299] Negotiation successful
0811 11:15:23.407456 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":161,"threads_started":1}
I20260811 11:15:23.413491 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.410762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:52195 (local address 127.28.25.1:32859)
0811 11:15:23.410831 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:23.410833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:23.410859 (+ 26us) server_negotiation.cc:407] Connection header received
0811 11:15:23.410949 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:23.410950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:23.410995 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:23.411070 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:23.411235 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.411729 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.412660 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.412903 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.412980 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.413192 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:23.413239 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:23.413306 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:23.413381 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 11:15:23.413406 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:15:23.413514 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.410553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:32859 (local address 127.28.25.3:52195)
0811 11:15:23.410793 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:23.410808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:23.410878 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:23.411075 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:23.411077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:23.411078 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:15:23.411163 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:23.411165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.411913 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.411914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:23.412564 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:23.412566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.413010 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.413011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:23.413087 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.413105 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:15:23.413319 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:15:23.413321 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 11:15:23.413375 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 11:15:23.413426 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":126,"threads_started":1}
I20260811 11:15:23.416482 30874 raft_consensus.cc:1261] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Refusing update from remote peer 9040217d356e4373b2fa651d958dcc6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:23.416460 31032 raft_consensus.cc:1261] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Refusing update from remote peer 9040217d356e4373b2fa651d958dcc6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:23.416886 31380 consensus_queue.cc:1035] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:23.417053 31374 consensus_queue.cc:1035] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:23.425088 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.422375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:44019 (local address 127.28.25.1:34725)
0811 11:15:23.422522 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:23.422533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:23.422622 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:23.422752 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:23.422752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:23.422754 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:15:23.422810 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:23.422812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.423564 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.423564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:23.424177 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:23.424179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.424636 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.424637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:23.424700 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.424705 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:15:23.424884 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:15:23.424886 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 11:15:23.424952 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 11:15:23.425007 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":95,"threads_started":1}
I20260811 11:15:23.425086 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.422444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:34725 (local address 127.28.25.3:44019)
0811 11:15:23.422518 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:23.422520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:23.422608 (+ 88us) server_negotiation.cc:407] Connection header received
0811 11:15:23.422678 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:23.422680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:23.422691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:23.422748 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:23.422864 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.423449 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.424269 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.424563 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.424633 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.424787 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:23.424823 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:23.424876 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:23.424958 (+ 82us) server_negotiation.cc:299] Negotiation successful
0811 11:15:23.424994 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:15:23.432358 30696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:23.432469 30696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:23.433264 30696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927: Sending DeleteTablet for 3 replicas of tablet 1097b8ee1a6a491fab81345af4196483
I20260811 11:15:23.433564 30851 tablet_service.cc:1514] Processing DeleteTablet for tablet 1097b8ee1a6a491fab81345af4196483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:23 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:46134
I20260811 11:15:23.433579 31001 tablet_service.cc:1514] Processing DeleteTablet for tablet 1097b8ee1a6a491fab81345af4196483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:23 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:32878
I20260811 11:15:23.433578 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 1097b8ee1a6a491fab81345af4196483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:23 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:49932
I20260811 11:15:23.433668 31358 tablet_replica.cc:330] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d: stopping tablet replica
I20260811 11:15:23.433668 31357 tablet_replica.cc:330] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d: stopping tablet replica
I20260811 11:15:23.433681 31356 tablet_replica.cc:330] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b: stopping tablet replica
I20260811 11:15:23.433718 31358 raft_consensus.cc:2229] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:23.433725 31356 raft_consensus.cc:2229] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:15:23.433724 31357 raft_consensus.cc:2229] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:23.433761 31358 raft_consensus.cc:2258] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:23.433772 31357 raft_consensus.cc:2258] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:23.433808 31356 raft_consensus.cc:2258] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:23.434230 31356 ts_tablet_manager.cc:1905] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:23.434247 31358 ts_tablet_manager.cc:1905] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:23.434262 30696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:15:23.434448 30696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:23.436180 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 7be35e12f1a245e59f77b972682de31d (DEFAULT_TABLE table=test-table [id=cf2481ee8ae54433a62e3fbecdd9293d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:23.436529 31357 ts_tablet_manager.cc:1905] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:23.436642 31001 tablet_service.cc:1467] Processing CreateTablet for tablet 7be35e12f1a245e59f77b972682de31d (DEFAULT_TABLE table=test-table [id=cf2481ee8ae54433a62e3fbecdd9293d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:23.436740 31001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be35e12f1a245e59f77b972682de31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:23.438304 31357 ts_tablet_manager.cc:1918] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:15:23.438391 31357 log.cc:1196] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/1097b8ee1a6a491fab81345af4196483
I20260811 11:15:23.438328 31368 tablet_bootstrap.cc:492] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d: Bootstrap starting.
I20260811 11:15:23.438673 31357 ts_tablet_manager.cc:1939] T 1097b8ee1a6a491fab81345af4196483 P fd7bb18d26df4a76bb342bc5d330d60d: Deleting consensus metadata
I20260811 11:15:23.438879 31368 tablet_bootstrap.cc:654] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:23.439332 30679 catalog_manager.cc:4841] TS fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293): tablet 1097b8ee1a6a491fab81345af4196483 (table test-table [id=9aaee3ef89f9490cb87bfefb2a30e533]) successfully deleted
I20260811 11:15:23.439457 30851 tablet_service.cc:1467] Processing CreateTablet for tablet 7be35e12f1a245e59f77b972682de31d (DEFAULT_TABLE table=test-table [id=cf2481ee8ae54433a62e3fbecdd9293d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:23.439680 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be35e12f1a245e59f77b972682de31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:23.439769 31358 ts_tablet_manager.cc:1918] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:15:23.439826 31358 log.cc:1196] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/1097b8ee1a6a491fab81345af4196483
I20260811 11:15:23.440060 31358 ts_tablet_manager.cc:1939] T 1097b8ee1a6a491fab81345af4196483 P 26abc7b9c0734f31b2bb916adc9aea1d: Deleting consensus metadata
I20260811 11:15:23.440202 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be35e12f1a245e59f77b972682de31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:23.441752 31367 tablet_bootstrap.cc:492] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d: Bootstrap starting.
I20260811 11:15:23.441879 31369 tablet_bootstrap.cc:492] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b: Bootstrap starting.
I20260811 11:15:23.442171 31367 tablet_bootstrap.cc:654] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:23.443113 31369 tablet_bootstrap.cc:654] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:23.443117 31367 tablet_bootstrap.cc:492] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d: No bootstrap required, opened a new log
I20260811 11:15:23.443176 31367 ts_tablet_manager.cc:1397] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:23.443394 31367 raft_consensus.cc:348] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.443521 31367 raft_consensus.cc:374] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:23.443557 31367 raft_consensus.cc:729] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26abc7b9c0734f31b2bb916adc9aea1d, State: Initialized, Role: FOLLOWER
I20260811 11:15:23.443615 31367 consensus_queue.cc:260] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.443768 31367 ts_tablet_manager.cc:1428] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:23.444396 31368 tablet_bootstrap.cc:492] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d: No bootstrap required, opened a new log
I20260811 11:15:23.444448 31368 ts_tablet_manager.cc:1397] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260811 11:15:23.444623 31368 raft_consensus.cc:348] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.444694 31368 raft_consensus.cc:374] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:23.444715 31368 raft_consensus.cc:729] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7bb18d26df4a76bb342bc5d330d60d, State: Initialized, Role: FOLLOWER
I20260811 11:15:23.444777 31368 consensus_queue.cc:260] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.444907 31368 ts_tablet_manager.cc:1428] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:23.445334 31369 tablet_bootstrap.cc:492] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b: No bootstrap required, opened a new log
I20260811 11:15:23.445406 31369 ts_tablet_manager.cc:1397] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260811 11:15:23.445890 31369 raft_consensus.cc:348] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.445983 31369 raft_consensus.cc:374] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:23.446010 31369 raft_consensus.cc:729] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040217d356e4373b2fa651d958dcc6b, State: Initialized, Role: FOLLOWER
I20260811 11:15:23.446086 31369 consensus_queue.cc:260] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.446341 31369 ts_tablet_manager.cc:1428] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260811 11:15:23.439680 31356 ts_tablet_manager.cc:1918] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:15:23.446519 31356 log.cc:1196] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/1097b8ee1a6a491fab81345af4196483
I20260811 11:15:23.446818 31356 ts_tablet_manager.cc:1939] T 1097b8ee1a6a491fab81345af4196483 P 9040217d356e4373b2fa651d958dcc6b: Deleting consensus metadata
I20260811 11:15:23.447358 30680 catalog_manager.cc:4841] TS 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019): tablet 1097b8ee1a6a491fab81345af4196483 (table test-table [id=9aaee3ef89f9490cb87bfefb2a30e533]) successfully deleted
I20260811 11:15:23.447562 30679 catalog_manager.cc:4841] TS 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859): tablet 1097b8ee1a6a491fab81345af4196483 (table test-table [id=9aaee3ef89f9490cb87bfefb2a30e533]) successfully deleted
I20260811 11:15:23.620532 31375 raft_consensus.cc:482] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:23.620672 31375 raft_consensus.cc:504] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.620931 31375 leader_election.cc:288] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859), 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019)
I20260811 11:15:23.621251 30874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be35e12f1a245e59f77b972682de31d" candidate_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" is_pre_election: true
I20260811 11:15:23.621243 31191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be35e12f1a245e59f77b972682de31d" candidate_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040217d356e4373b2fa651d958dcc6b" is_pre_election: true
I20260811 11:15:23.621371 31191 raft_consensus.cc:2454] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd7bb18d26df4a76bb342bc5d330d60d in term 0.
I20260811 11:15:23.621371 30874 raft_consensus.cc:2454] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd7bb18d26df4a76bb342bc5d330d60d in term 0.
I20260811 11:15:23.621557 30957 leader_election.cc:302] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9040217d356e4373b2fa651d958dcc6b, fd7bb18d26df4a76bb342bc5d330d60d; no voters:
I20260811 11:15:23.621699 31375 raft_consensus.cc:2781] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:23.621791 31375 raft_consensus.cc:482] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:23.621825 31375 raft_consensus.cc:3037] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:23.622725 31375 raft_consensus.cc:504] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.622926 31375 leader_election.cc:288] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [CANDIDATE]: Term 1 election: Requested vote from peers 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859), 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019)
I20260811 11:15:23.623071 30874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be35e12f1a245e59f77b972682de31d" candidate_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26abc7b9c0734f31b2bb916adc9aea1d"
I20260811 11:15:23.623171 30874 raft_consensus.cc:3037] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:23.623136 31191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be35e12f1a245e59f77b972682de31d" candidate_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040217d356e4373b2fa651d958dcc6b"
I20260811 11:15:23.623219 31191 raft_consensus.cc:3037] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:23.624251 30874 raft_consensus.cc:2454] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd7bb18d26df4a76bb342bc5d330d60d in term 1.
I20260811 11:15:23.624251 31191 raft_consensus.cc:2454] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd7bb18d26df4a76bb342bc5d330d60d in term 1.
I20260811 11:15:23.624467 30957 leader_election.cc:302] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9040217d356e4373b2fa651d958dcc6b, fd7bb18d26df4a76bb342bc5d330d60d; no voters:
I20260811 11:15:23.624588 31375 raft_consensus.cc:2781] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:23.624740 31375 raft_consensus.cc:686] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 1 LEADER]: Becoming Leader. State: Replica: fd7bb18d26df4a76bb342bc5d330d60d, State: Running, Role: LEADER
I20260811 11:15:23.624804 31375 consensus_queue.cc:237] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } }
I20260811 11:15:23.625682 30695 catalog_manager.cc:5495] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d reported cstate change: term changed from 0 to 1, leader changed from <none> to fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2). New cstate: current_term: 1 leader_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd7bb18d26df4a76bb342bc5d330d60d" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 34293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 } health_report { overall_health: UNKNOWN } } }
I20260811 11:15:23.671916 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:23.669548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.28.25.2:34293)
0811 11:15:23.669619 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:23.669621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:23.669631 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:15:23.669660 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:23.669661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:23.669675 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:23.669728 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:23.669846 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.670350 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.671143 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:23.671369 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:23.671452 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:23.671617 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:23.671655 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:23.671721 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:23.671797 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 11:15:23.671824 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:15:23.674834 31191 raft_consensus.cc:1261] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Refusing update from remote peer fd7bb18d26df4a76bb342bc5d330d60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:23.674870 30874 raft_consensus.cc:1261] T 7be35e12f1a245e59f77b972682de31d P 26abc7b9c0734f31b2bb916adc9aea1d [term 1 FOLLOWER]: Refusing update from remote peer fd7bb18d26df4a76bb342bc5d330d60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:23.675146 31399 consensus_queue.cc:1035] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040217d356e4373b2fa651d958dcc6b" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:23.675278 31375 consensus_queue.cc:1035] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "26abc7b9c0734f31b2bb916adc9aea1d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 32859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 11:15:23.680864 31340 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 11:15:25.203907 30695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:25.204032 30695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:25.205071 30695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe4f0a88d4224922ad9268fc460c9927: Sending DeleteTablet for 3 replicas of tablet 7be35e12f1a245e59f77b972682de31d
I20260811 11:15:25.205386 31001 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be35e12f1a245e59f77b972682de31d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:25 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:32878
I20260811 11:15:25.205390 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be35e12f1a245e59f77b972682de31d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:25 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:49932
I20260811 11:15:25.205538 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 30777
I20260811 11:15:25.205616 30851 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be35e12f1a245e59f77b972682de31d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:25 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:46134
I20260811 11:15:25.209980 31432 tablet_replica.cc:330] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b: stopping tablet replica
I20260811 11:15:25.210083 31432 raft_consensus.cc:2229] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:25.210147 31432 raft_consensus.cc:2258] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:25.210295 31431 tablet_replica.cc:330] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d: stopping tablet replica
I20260811 11:15:25.210445 31431 raft_consensus.cc:2229] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:15:25.210541 31432 ts_tablet_manager.cc:1905] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:25.210570 31431 raft_consensus.cc:2258] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:25.210896 31431 ts_tablet_manager.cc:1905] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:25.212605 31432 ts_tablet_manager.cc:1918] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 11:15:25.212711 31432 log.cc:1196] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/7be35e12f1a245e59f77b972682de31d
I20260811 11:15:25.213001 31432 ts_tablet_manager.cc:1939] T 7be35e12f1a245e59f77b972682de31d P 9040217d356e4373b2fa651d958dcc6b: Deleting consensus metadata
I20260811 11:15:25.213068 31431 ts_tablet_manager.cc:1918] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 11:15:25.213145 31431 log.cc:1196] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/7be35e12f1a245e59f77b972682de31d
I20260811 11:15:25.213469 30680 catalog_manager.cc:4841] TS 9040217d356e4373b2fa651d958dcc6b (127.28.25.3:44019): tablet 7be35e12f1a245e59f77b972682de31d (table test-table [id=cf2481ee8ae54433a62e3fbecdd9293d]) successfully deleted
I20260811 11:15:25.213569 31431 ts_tablet_manager.cc:1939] T 7be35e12f1a245e59f77b972682de31d P fd7bb18d26df4a76bb342bc5d330d60d: Deleting consensus metadata
I20260811 11:15:25.213945 30679 catalog_manager.cc:4841] TS fd7bb18d26df4a76bb342bc5d330d60d (127.28.25.2:34293): tablet 7be35e12f1a245e59f77b972682de31d (table test-table [id=cf2481ee8ae54433a62e3fbecdd9293d]) successfully deleted
W20260811 11:15:25.216384 30679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:32859 (error 108)
I20260811 11:15:25.216689 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 30936
I20260811 11:15:25.223315 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:25.216860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:32859 (local address 127.0.0.1:46164)
0811 11:15:25.223174 (+ 6314us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:25.223261 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:32859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6269,"thread_start_us":69,"threads_started":1}
W20260811 11:15:25.223582 30679 catalog_manager.cc:4584] TS 26abc7b9c0734f31b2bb916adc9aea1d (127.28.25.1:32859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7be35e12f1a245e59f77b972682de31d: Network error: Client connection negotiation failed: client connection to 127.28.25.1:32859: connect: Connection refused (error 111)
I20260811 11:15:25.227695 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 31093
I20260811 11:15:25.236974 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 30660
2026-08-11T11:15:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6174 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:25 dist-test-slave-2w3w krb5kdc[31438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:25 dist-test-slave-2w3w krb5kdc[31438](info): set up 2 sockets
Aug 11 11:15:25 dist-test-slave-2w3w krb5kdc[31438](info): commencing operation
krb5kdc: starting...
W20260811 11:15:26.852115 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.590s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:26 dist-test-slave-2w3w krb5kdc[31438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:15:27.034507 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:43977
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:43977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:27.129607 31452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:27.129885 31452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:27.129940 31452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:27.131302 31452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:27.131387 31452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:27.131413 31452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:27.131436 31452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:27.131467 31452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:27.133141 31452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:43977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:43977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:27.133460 31452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:27.133728 31452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:27.140066 31458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:27.140154 31457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:27.140265 31452 server_base.cc:1034] running on GCE node
W20260811 11:15:27.140074 31460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:27.140650 31452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:27.141009 31452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:27.142195 31452 hybrid_clock.cc:648] HybridClock initialized: now 1786446927142170 us; error 45 us; skew 500 ppm
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[31438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:27.145355 31452 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:15:27.145911 31452 webserver.cc:460] Webserver started at http://127.28.25.62:43123/ using document root <none> and password file <none>
I20260811 11:15:27.146221 31452 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:27.146311 31452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:27.146495 31452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:27.147881 31452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "26cc0d5dbd53af589504ddb5d7cf3cd6"
server_key_iv: "6ce178b6eaacb43c5771ed5657b77874"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.148304 31452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "26cc0d5dbd53af589504ddb5d7cf3cd6"
server_key_iv: "6ce178b6eaacb43c5771ed5657b77874"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.150104 31452 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260811 11:15:27.151260 31467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:27.151541 31452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:27.151666 31452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "26cc0d5dbd53af589504ddb5d7cf3cd6"
server_key_iv: "6ce178b6eaacb43c5771ed5657b77874"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.151768 31452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:27.181497 31452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:27.184193 31452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:27.184386 31452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:27.188809 31452 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:43977
I20260811 11:15:27.188959 31527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:43977 every 8 connection(s)
I20260811 11:15:27.189306 31452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:27.190292 31528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:27.191164 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 31452
I20260811 11:15:27.191241 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:27.191443 28772 external_mini_cluster.cc:1428] Setting key 0ce6277797798572bf2ef79ffde516fc
I20260811 11:15:27.194072 31528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014: Bootstrap starting.
I20260811 11:15:27.195024 31528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:27.195390 31528 log.cc:824] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:27.196332 31528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014: No bootstrap required, opened a new log
I20260811 11:15:27.198472 31528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43977 } }
I20260811 11:15:27.198668 31528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:27.198714 31528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014, State: Initialized, Role: FOLLOWER
I20260811 11:15:27.198889 31528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43977 } }
I20260811 11:15:27.199023 31528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:27.199076 31528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:27.199133 31528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [term 0 FOLLOWER]: Advancing to term 1
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[31438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:27.199894 31528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43977 } }
I20260811 11:15:27.200037 31528 leader_election.cc:302] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014; no voters:
I20260811 11:15:27.200268 31528 leader_election.cc:288] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:27.200341 31535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:27.200594 31535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [term 1 LEADER]: Becoming Leader. State: Replica: 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014, State: Running, Role: LEADER
I20260811 11:15:27.200621 31528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:27.202181 31535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43977 } }
I20260811 11:15:27.203011 31537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43977 } } }
I20260811 11:15:27.203025 31538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014. Latest consensus state: current_term: 1 leader_uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd3ef3fb2bd4b5d9d2ed53c7d5ee014" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43977 } } }
I20260811 11:15:27.203190 31537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:27.203197 31538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014 [sys.catalog]: This master's current role is: LEADER
W20260811 11:15:27.203462 31550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:15:27.203579 31550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:15:27.203685 31552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:27.203924 31552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:27.204365 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:27.192098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.28.25.62:43977)
0811 11:15:27.192340 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:27.192352 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:27.192395 (+ 43us) server_negotiation.cc:407] Connection header received
0811 11:15:27.193103 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:27.193114 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:27.193192 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:27.193377 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:27.194028 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.195678 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.196622 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.197069 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.197202 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:27.199873 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:27.199878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:27.199878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:27.199894 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:27.202221 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:27.202857 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:27.202862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:27.202863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:27.203042 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:27.203338 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:27.203339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:27.203339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:27.203698 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:27.203923 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:27.204021 (+ 98us) server_negotiation.cc:299] Negotiation successful
0811 11:15:27.204178 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260811 11:15:27.205996 31552 catalog_manager.cc:1347] Generated new cluster ID: 813728b0e4644c3d8d8c7c5c06c49567
I20260811 11:15:27.206059 31552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:27.223831 31552 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:27.224486 31552 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:27.228044 31552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bd3ef3fb2bd4b5d9d2ed53c7d5ee014: Generated new TSK 0
I20260811 11:15:27.228333 31552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:15:27.271313 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43977
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:27.387141 31557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:27.387384 31557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:27.387413 31557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:27.388728 31557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:27.388784 31557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:27.388815 31557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:27.390849 31557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:27.391206 31557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:27.391597 31557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:15:27.397773 31557 server_base.cc:1034] running on GCE node
W20260811 11:15:27.397759 31562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:27.397917 31563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:27.397761 31565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:27.398409 31557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:27.398706 31557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:27.399878 31557 hybrid_clock.cc:648] HybridClock initialized: now 1786446927399875 us; error 48 us; skew 500 ppm
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[31438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:27.403298 31557 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:15:27.403859 31557 webserver.cc:460] Webserver started at http://127.28.25.1:35569/ using document root <none> and password file <none>
I20260811 11:15:27.404145 31557 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:27.404223 31557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:27.404408 31557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:27.405695 31557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "9012e8b2cdfe46909419f1df79cf3f72"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "e3dd5ac99416948e153a24def2bfb10a"
server_key_iv: "202fdd655698bf1d4bc9a8e7d3c4bf13"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.406126 31557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "9012e8b2cdfe46909419f1df79cf3f72"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "e3dd5ac99416948e153a24def2bfb10a"
server_key_iv: "202fdd655698bf1d4bc9a8e7d3c4bf13"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.407964 31557 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260811 11:15:27.409139 31574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:27.409333 31557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:27.409442 31557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "9012e8b2cdfe46909419f1df79cf3f72"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "e3dd5ac99416948e153a24def2bfb10a"
server_key_iv: "202fdd655698bf1d4bc9a8e7d3c4bf13"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.409528 31557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:27.435508 31557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:27.438333 31557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:27.438531 31557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:27.438845 31557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:27.439242 31557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:27.439303 31557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:27.439349 31557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:27.439405 31557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:27.446267 31557 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:44289
I20260811 11:15:27.446344 31699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:44289 every 8 connection(s)
I20260811 11:15:27.446768 31557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:27.449273 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 31557
I20260811 11:15:27.449363 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:27.449599 28772 external_mini_cluster.cc:1428] Setting key c9f770e3be3cbea43f100ef4d8959b20
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[31438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:27.458796 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:27.447662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:58919 (local address 127.28.25.62:43977)
0811 11:15:27.447725 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:27.447727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:27.447966 (+ 239us) server_negotiation.cc:407] Connection header received
0811 11:15:27.448439 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:27.448440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:27.448451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:27.448522 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:27.449611 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.450246 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.451480 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.451829 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.451908 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:27.454885 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:27.454886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:27.454887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:27.454895 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:27.456653 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:27.457229 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:27.457229 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:27.457230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:27.457283 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:27.457695 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:27.457696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:27.457696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:27.457986 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:27.458080 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:27.458628 (+ 548us) server_negotiation.cc:299] Negotiation successful
0811 11:15:27.458723 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 11:15:27.459128 31703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:27.447608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:43977 (local address 127.28.25.1:58919)
0811 11:15:27.447788 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:27.447835 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:27.448348 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:27.448580 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:27.448583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:27.449199 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:27.449495 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:27.449502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.450408 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.450409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:27.451361 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:27.451364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.451935 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.451936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:27.452068 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:27.452807 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:27.452825 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:27.454733 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:27.456753 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:27.456764 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:27.456767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:27.457147 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:27.457367 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:27.457368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:27.457368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:27.457551 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:27.458089 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:27.458098 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:27.458436 (+ 338us) client_negotiation.cc:769] Sending connection context
0811 11:15:27.458594 (+ 158us) client_negotiation.cc:240] Negotiation successful
0811 11:15:27.458802 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260811 11:15:27.459653 31701 heartbeater.cc:347] Connected to a master server at 127.28.25.62:43977
I20260811 11:15:27.459784 31701 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:27.460024 31701 heartbeater.cc:511] Master 127.28.25.62:43977 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
I20260811 11:15:27.460675 31487 ts_manager.cc:194] Registered new tserver with Master: 9012e8b2cdfe46909419f1df79cf3f72 (127.28.25.1:44289)
I20260811 11:15:27.461325 31487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:58919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:15:27.483654 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43977
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:27.570106 31706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:27.570343 31706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:27.570370 31706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:27.571671 31706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:27.571725 31706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:27.571754 31706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:27.573324 31706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:27.573607 31706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:27.573872 31706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:27.579324 31711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:27.579439 31712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:27.579561 31714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:27.579527 31706 server_base.cc:1034] running on GCE node
I20260811 11:15:27.579828 31706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:27.580176 31706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:27.581403 31706 hybrid_clock.cc:648] HybridClock initialized: now 1786446927581390 us; error 59 us; skew 500 ppm
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[31438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:27.584314 31706 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:15:27.584878 31706 webserver.cc:460] Webserver started at http://127.28.25.2:44779/ using document root <none> and password file <none>
I20260811 11:15:27.585139 31706 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:27.585209 31706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:27.585364 31706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:27.586509 31706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "819b6378e53441c88fada0a616b6186f"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "47f494a536ec724be9f89f9ac765b9dd"
server_key_iv: "f4053d63c484e36f92a2f816e9f620ac"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.586913 31706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "819b6378e53441c88fada0a616b6186f"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "47f494a536ec724be9f89f9ac765b9dd"
server_key_iv: "f4053d63c484e36f92a2f816e9f620ac"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.588469 31706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:27.589452 31722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:27.590075 31706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:27.590236 31706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "819b6378e53441c88fada0a616b6186f"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "47f494a536ec724be9f89f9ac765b9dd"
server_key_iv: "f4053d63c484e36f92a2f816e9f620ac"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.590436 31706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:27.603753 31706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:27.606287 31706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:27.606501 31706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:27.606784 31706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:27.607298 31706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:27.607362 31706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:27.607409 31706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:27.607434 31706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:27.613935 31706 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:33615
I20260811 11:15:27.614029 31851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:33615 every 8 connection(s)
I20260811 11:15:27.614394 31706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:15:27.620810 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 31706
I20260811 11:15:27.620918 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:27.621133 28772 external_mini_cluster.cc:1428] Setting key 6ddebe8f1cc65861c3d2b5b0ed4f93f7
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[31438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:27.625728 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:27.615296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:48741 (local address 127.28.25.62:43977)
0811 11:15:27.615368 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:27.615370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:27.615621 (+ 251us) server_negotiation.cc:407] Connection header received
0811 11:15:27.616177 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:27.616179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:27.616190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:27.616272 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:27.617289 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.617875 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.618925 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.619162 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.619224 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:27.622069 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:27.622070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:27.622071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:27.622080 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:27.623939 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:27.624567 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:27.624568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:27.624569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:27.624635 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:27.624969 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:27.624969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:27.624970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:27.625096 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:27.625153 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:27.625568 (+ 415us) server_negotiation.cc:299] Negotiation successful
0811 11:15:27.625656 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 11:15:27.626030 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:27.615279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:43977 (local address 127.28.25.2:48741)
0811 11:15:27.615462 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:27.615503 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:27.616073 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:27.616328 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:27.616331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:27.616940 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:27.617176 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:27.617181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.617983 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.617984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:27.618820 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:27.618823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.619275 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.619276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:27.619400 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:27.620039 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:27.620048 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:27.621956 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:27.624045 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:27.624058 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:27.624061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:27.624463 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:27.624718 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:27.624719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:27.624720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:27.624893 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:27.625170 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:27.625180 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:27.625407 (+ 227us) client_negotiation.cc:769] Sending connection context
0811 11:15:27.625532 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 11:15:27.625737 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
I20260811 11:15:27.626717 31852 heartbeater.cc:347] Connected to a master server at 127.28.25.62:43977
I20260811 11:15:27.626859 31852 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:27.627134 31852 heartbeater.cc:511] Master 127.28.25.62:43977 requested a full tablet report, sending...
I20260811 11:15:27.627652 31487 ts_manager.cc:194] Registered new tserver with Master: 819b6378e53441c88fada0a616b6186f (127.28.25.2:33615)
I20260811 11:15:27.628122 31487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:48741
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:15:27.655121 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43977
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:27.752797 31858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:27.753052 31858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:27.753080 31858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:27.754490 31858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:27.754547 31858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:27.754580 31858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:27.756446 31858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:27.756683 31858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:27.756891 31858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:27.762861 31864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:27.762880 31868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:27.762861 31866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:27.763357 31858 server_base.cc:1034] running on GCE node
I20260811 11:15:27.763588 31858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:27.763871 31858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:27.765069 31858 hybrid_clock.cc:648] HybridClock initialized: now 1786446927765039 us; error 43 us; skew 500 ppm
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[31438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:27.768321 31858 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:15:27.768895 31858 webserver.cc:460] Webserver started at http://127.28.25.3:38535/ using document root <none> and password file <none>
I20260811 11:15:27.769184 31858 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:27.769236 31858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:27.769428 31858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:27.770634 31858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "3d67b6bc32904e1285c988c2a8d46bd8"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "fac9780ba36d75e5aa75eb355470fc59"
server_key_iv: "97c69558215ddcc2dec16c57f66f66c9"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.771076 31858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "3d67b6bc32904e1285c988c2a8d46bd8"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "fac9780ba36d75e5aa75eb355470fc59"
server_key_iv: "97c69558215ddcc2dec16c57f66f66c9"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.772778 31858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:27.773663 31875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:27.773918 31858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:27.774026 31858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "3d67b6bc32904e1285c988c2a8d46bd8"
format_stamp: "Formatted at 2026-08-11 11:15:27 on dist-test-slave-2w3w"
server_key: "fac9780ba36d75e5aa75eb355470fc59"
server_key_iv: "97c69558215ddcc2dec16c57f66f66c9"
server_key_version: "encryptionkey@0"
I20260811 11:15:27.774125 31858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:27.794641 31858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:27.797407 31858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:27.797619 31858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:27.797948 31858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:27.798318 31858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:27.798378 31858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:27.798424 31858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:27.798447 31858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:27.805706 31858 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:40589
I20260811 11:15:27.805764 32002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:40589 every 8 connection(s)
I20260811 11:15:27.806169 31858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:15:27.812819 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 31858
I20260811 11:15:27.812961 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[31438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:27.813182 28772 external_mini_cluster.cc:1428] Setting key d0e3522189475fcf805fc11f7e5ad673
I20260811 11:15:27.816951 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:27.807089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:49191 (local address 127.28.25.62:43977)
0811 11:15:27.807167 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:27.807169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:27.807444 (+ 275us) server_negotiation.cc:407] Connection header received
0811 11:15:27.808019 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:27.808021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:27.808035 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:27.808091 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:27.809164 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.809691 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.810783 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.811067 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.811145 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:27.813777 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:27.813779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:27.813780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:27.813789 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:27.815394 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:27.815862 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:27.815863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:27.815864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:27.815941 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:27.816218 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:27.816219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:27.816220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:27.816361 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:27.816446 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:27.816747 (+ 301us) server_negotiation.cc:299] Negotiation successful
0811 11:15:27.816840 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:15:27.817198 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:27.807080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:43977 (local address 127.28.25.3:49191)
0811 11:15:27.807277 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:27.807319 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:27.807900 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:27.808174 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:27.808178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:27.808809 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:27.809048 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:27.809054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.809810 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.809811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:27.810678 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:27.810680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:27.811160 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:27.811161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:27.811312 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:27.811932 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:27.811941 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:27.813669 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:27.815504 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:27.815514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:27.815517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:27.815777 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:27.816021 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:27.816022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:27.816023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:27.816149 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:27.816435 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:27.816443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:27.816609 (+ 166us) client_negotiation.cc:769] Sending connection context
0811 11:15:27.816725 (+ 116us) client_negotiation.cc:240] Negotiation successful
0811 11:15:27.816915 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260811 11:15:27.817790 32003 heartbeater.cc:347] Connected to a master server at 127.28.25.62:43977
I20260811 11:15:27.817893 32003 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:27.818121 32003 heartbeater.cc:511] Master 127.28.25.62:43977 requested a full tablet report, sending...
I20260811 11:15:27.818611 31487 ts_manager.cc:194] Registered new tserver with Master: 3d67b6bc32904e1285c988c2a8d46bd8 (127.28.25.3:40589)
I20260811 11:15:27.819154 31487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:49191
I20260811 11:15:27.825313 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 11:15:27.839877 31533 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:15:27.838248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.28.25.62:43977)
0811 11:15:27.838358 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:27.838360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:27.838562 (+ 202us) server_negotiation.cc:407] Connection header received
0811 11:15:27.838597 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:27.838598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:27.838610 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:27.838659 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:27.839754 (+ 1095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60716: BlockingRecv error: recv got EOF from 127.0.0.1:60716 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 11:15:27.840888 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 31557
I20260811 11:15:27.848543 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 31706
I20260811 11:15:27.863984 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 31858
I20260811 11:15:27.874462 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 31452
2026-08-11T11:15:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2637 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[32017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[32017](info): set up 2 sockets
Aug 11 11:15:27 dist-test-slave-2w3w krb5kdc[32017](info): commencing operation
krb5kdc: starting...
W20260811 11:15:29.010329 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:29 dist-test-slave-2w3w krb5kdc[32017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:15:29.155117 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:38411
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:29.241190 32031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:29.241493 32031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:29.241539 32031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:29.242851 32031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:29.242935 32031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:29.242959 32031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:29.242977 32031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:29.243064 32031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:29.244817 32031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:29.245151 32031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:29.245419 32031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:29.250782 32037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:29.250859 32036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:29.250782 32039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:29.251076 32031 server_base.cc:1034] running on GCE node
I20260811 11:15:29.251350 32031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:29.251662 32031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:29.252887 32031 hybrid_clock.cc:648] HybridClock initialized: now 1786446929252878 us; error 42 us; skew 500 ppm
Aug 11 11:15:29 dist-test-slave-2w3w krb5kdc[32017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446929, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:29.255673 32031 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:15:29.256268 32031 webserver.cc:460] Webserver started at http://127.28.25.62:34787/ using document root <none> and password file <none>
I20260811 11:15:29.256541 32031 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:29.256615 32031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:29.256773 32031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:29.258001 32031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "b615fd0de9e34ce5947a9929fc87ec43"
format_stamp: "Formatted at 2026-08-11 11:15:29 on dist-test-slave-2w3w"
server_key: "41dbe53b02f2d2e0d7b6accce155f61c"
server_key_iv: "15d7bc1f92f8386a306a6721da962899"
server_key_version: "encryptionkey@0"
I20260811 11:15:29.258386 32031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "b615fd0de9e34ce5947a9929fc87ec43"
format_stamp: "Formatted at 2026-08-11 11:15:29 on dist-test-slave-2w3w"
server_key: "41dbe53b02f2d2e0d7b6accce155f61c"
server_key_iv: "15d7bc1f92f8386a306a6721da962899"
server_key_version: "encryptionkey@0"
I20260811 11:15:29.260025 32031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:15:29.260969 32047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:29.261255 32031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:29.261384 32031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "b615fd0de9e34ce5947a9929fc87ec43"
format_stamp: "Formatted at 2026-08-11 11:15:29 on dist-test-slave-2w3w"
server_key: "41dbe53b02f2d2e0d7b6accce155f61c"
server_key_iv: "15d7bc1f92f8386a306a6721da962899"
server_key_version: "encryptionkey@0"
I20260811 11:15:29.261478 32031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:29.275283 32031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:29.277812 32031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:29.277989 32031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:29.282543 32031 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:38411
I20260811 11:15:29.282608 32106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:38411 every 8 connection(s)
I20260811 11:15:29.283306 32031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:29.284107 32107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:29.286650 32107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43: Bootstrap starting.
I20260811 11:15:29.287388 32107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:29.287730 32107 log.cc:824] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:29.288591 32107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43: No bootstrap required, opened a new log
I20260811 11:15:29.290393 32107 raft_consensus.cc:348] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615fd0de9e34ce5947a9929fc87ec43" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38411 } }
I20260811 11:15:29.290550 32107 raft_consensus.cc:374] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:29.290573 32107 raft_consensus.cc:729] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b615fd0de9e34ce5947a9929fc87ec43, State: Initialized, Role: FOLLOWER
I20260811 11:15:29.290741 32107 consensus_queue.cc:260] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615fd0de9e34ce5947a9929fc87ec43" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38411 } }
I20260811 11:15:29.290830 32107 raft_consensus.cc:388] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:29.290855 32107 raft_consensus.cc:482] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:29.290886 32107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:29.291476 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 32031
I20260811 11:15:29.291569 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:29.291776 28772 external_mini_cluster.cc:1428] Setting key 6bf1cf1128d8f8cafd9c86e6cb7fdc36
I20260811 11:15:29.292160 32107 raft_consensus.cc:504] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615fd0de9e34ce5947a9929fc87ec43" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38411 } }
I20260811 11:15:29.292578 32107 leader_election.cc:302] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b615fd0de9e34ce5947a9929fc87ec43; no voters:
I20260811 11:15:29.292831 32107 leader_election.cc:288] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:29.292870 32112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:29.292982 32112 raft_consensus.cc:686] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [term 1 LEADER]: Becoming Leader. State: Replica: b615fd0de9e34ce5947a9929fc87ec43, State: Running, Role: LEADER
I20260811 11:15:29.293146 32107 sys_catalog.cc:564] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:29.293141 32112 consensus_queue.cc:237] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615fd0de9e34ce5947a9929fc87ec43" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38411 } }
I20260811 11:15:29.293653 32114 sys_catalog.cc:455] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b615fd0de9e34ce5947a9929fc87ec43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615fd0de9e34ce5947a9929fc87ec43" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38411 } } }
I20260811 11:15:29.293749 32114 sys_catalog.cc:458] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:29.294057 32113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b615fd0de9e34ce5947a9929fc87ec43. Latest consensus state: current_term: 1 leader_uuid: "b615fd0de9e34ce5947a9929fc87ec43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615fd0de9e34ce5947a9929fc87ec43" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38411 } } }
I20260811 11:15:29.294261 32120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:29.294477 32113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:29.295032 32120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:29.296821 32120 catalog_manager.cc:1347] Generated new cluster ID: 2bbb0328df8f49b89f891e448fbc4bef
I20260811 11:15:29.296876 32120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 11:15:29 dist-test-slave-2w3w krb5kdc[32017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:29.303207 32120 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:29.303756 32120 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:29.304996 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:29.292380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.28.25.62:38411)
0811 11:15:29.292595 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:29.292604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:29.292636 (+ 32us) server_negotiation.cc:407] Connection header received
0811 11:15:29.292786 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:29.292797 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:29.292855 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:29.293952 (+ 1097us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:29.294086 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:29.295560 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:29.296655 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:29.297278 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:29.297390 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:29.300507 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:29.300511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:29.300511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:29.300524 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:29.302617 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:29.303069 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:29.303073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:29.303074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:29.303213 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:29.303476 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:29.303477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:29.303477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:29.303841 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:29.304010 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:29.304553 (+ 543us) server_negotiation.cc:299] Negotiation successful
0811 11:15:29.304728 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
I20260811 11:15:29.306080 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:15:29.308281 32120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b615fd0de9e34ce5947a9929fc87ec43: Generated new TSK 0
I20260811 11:15:29.308638 32120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:15:29.323361 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:29.318821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.28.25.62:38411)
0811 11:15:29.318880 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:29.318883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:29.319032 (+ 149us) server_negotiation.cc:407] Connection header received
0811 11:15:29.319136 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:29.319137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:29.319165 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:29.319232 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:29.319655 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:29.320518 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:29.321395 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:29.321761 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:29.321852 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:29.322725 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:29.322727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:29.322727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:29.323039 (+ 312us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:29.323061 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:29.323069 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:29.323134 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:29.323228 (+ 94us) server_negotiation.cc:299] Negotiation successful
0811 11:15:29.323264 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 11:15:29.326357 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 32031
2026-08-11T11:15:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1450 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:29 dist-test-slave-2w3w krb5kdc[32142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:29 dist-test-slave-2w3w krb5kdc[32142](info): set up 2 sockets
Aug 11 11:15:29 dist-test-slave-2w3w krb5kdc[32142](info): commencing operation
krb5kdc: starting...
W20260811 11:15:30.438620 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:30 dist-test-slave-2w3w krb5kdc[32142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:15:30.574605 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:40767
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:40767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:30.660055 32158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:30.660342 32158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:30.660386 32158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:30.661634 32158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:30.661721 32158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:30.661746 32158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:30.661767 32158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:30.661796 32158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:30.663652 32158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:40767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:30.663980 32158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:30.664228 32158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:30.669514 32168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:30.669502 32165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:30.669749 32158 server_base.cc:1034] running on GCE node
W20260811 11:15:30.669524 32166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:30.670198 32158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:30.670580 32158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:30.671758 32158 hybrid_clock.cc:648] HybridClock initialized: now 1786446930671751 us; error 40 us; skew 500 ppm
Aug 11 11:15:30 dist-test-slave-2w3w krb5kdc[32142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446930, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:30.681102 32158 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:15:30.681702 32158 webserver.cc:460] Webserver started at http://127.28.25.62:36441/ using document root <none> and password file <none>
I20260811 11:15:30.682020 32158 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:30.682102 32158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:30.682291 32158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:30.683641 32158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "852eada95ea44f079bbdfc934f6a851b"
format_stamp: "Formatted at 2026-08-11 11:15:30 on dist-test-slave-2w3w"
server_key: "905a7301abb7377d3e4714d14200516d"
server_key_iv: "e6858dddff3513b05f1ddde053736daa"
server_key_version: "encryptionkey@0"
I20260811 11:15:30.684266 32158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "852eada95ea44f079bbdfc934f6a851b"
format_stamp: "Formatted at 2026-08-11 11:15:30 on dist-test-slave-2w3w"
server_key: "905a7301abb7377d3e4714d14200516d"
server_key_iv: "e6858dddff3513b05f1ddde053736daa"
server_key_version: "encryptionkey@0"
I20260811 11:15:30.686131 32158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:15:30.687197 32177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:30.687480 32158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:30.687618 32158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "852eada95ea44f079bbdfc934f6a851b"
format_stamp: "Formatted at 2026-08-11 11:15:30 on dist-test-slave-2w3w"
server_key: "905a7301abb7377d3e4714d14200516d"
server_key_iv: "e6858dddff3513b05f1ddde053736daa"
server_key_version: "encryptionkey@0"
I20260811 11:15:30.687713 32158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:30.713462 32158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:30.715837 32158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:30.716070 32158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:30.720608 32158 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:40767
I20260811 11:15:30.720654 32235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:40767 every 8 connection(s)
I20260811 11:15:30.721053 32158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:30.721415 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 32158
I20260811 11:15:30.721524 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:30.721722 28772 external_mini_cluster.cc:1428] Setting key ba70592b819d1d57146d3efb682a7b47
I20260811 11:15:30.722683 32236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:30.726091 32236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b: Bootstrap starting.
I20260811 11:15:30.726873 32236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:30.727242 32236 log.cc:824] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:30.728263 32236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b: No bootstrap required, opened a new log
Aug 11 11:15:30 dist-test-slave-2w3w krb5kdc[32142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:30.730713 32236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852eada95ea44f079bbdfc934f6a851b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 40767 } }
I20260811 11:15:30.730942 32236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:30.731050 32236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852eada95ea44f079bbdfc934f6a851b, State: Initialized, Role: FOLLOWER
I20260811 11:15:30.731232 32236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852eada95ea44f079bbdfc934f6a851b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 40767 } }
I20260811 11:15:30.731309 32236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:30.731379 32236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:30.731426 32236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:30.732080 32236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852eada95ea44f079bbdfc934f6a851b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 40767 } }
I20260811 11:15:30.732172 32236 leader_election.cc:302] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 852eada95ea44f079bbdfc934f6a851b; no voters:
I20260811 11:15:30.732426 32236 leader_election.cc:288] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:30.732466 32241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:30.732594 32241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [term 1 LEADER]: Becoming Leader. State: Replica: 852eada95ea44f079bbdfc934f6a851b, State: Running, Role: LEADER
I20260811 11:15:30.732684 32241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852eada95ea44f079bbdfc934f6a851b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 40767 } }
I20260811 11:15:30.732743 32236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:30.733023 32241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 852eada95ea44f079bbdfc934f6a851b. Latest consensus state: current_term: 1 leader_uuid: "852eada95ea44f079bbdfc934f6a851b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852eada95ea44f079bbdfc934f6a851b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 40767 } } }
I20260811 11:15:30.733098 32241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:30.733291 32242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "852eada95ea44f079bbdfc934f6a851b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852eada95ea44f079bbdfc934f6a851b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 40767 } } }
I20260811 11:15:30.733385 32242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:30.733632 32245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:30.734258 32245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:30.734779 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:30.722446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.28.25.62:40767)
0811 11:15:30.722637 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:30.722646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:30.722674 (+ 28us) server_negotiation.cc:407] Connection header received
0811 11:15:30.723321 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:30.723329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:30.723387 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:30.723524 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:30.724495 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:30.725673 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:30.726760 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:30.727387 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:30.727484 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:30.730297 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:30.730302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:30.730303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:30.730314 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:30.732421 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:30.733032 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:30.733035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:30.733036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:30.733180 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:30.733518 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:30.733519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:30.733519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:30.733893 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:30.734065 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:30.734356 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 11:15:30.734509 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260811 11:15:30.736156 32245 catalog_manager.cc:1347] Generated new cluster ID: 7ffc55b3829f4d27a75e69e9d52c7930
I20260811 11:15:30.736212 32245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:30.743929 32245 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:30.744545 32245 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:30.749922 32245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 852eada95ea44f079bbdfc934f6a851b: Generated new TSK 0
I20260811 11:15:30.750134 32245 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:15:30.791455 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:40767
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:30.890421 32264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:30.890755 32264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:30.890785 32264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:30.892493 32264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:30.892556 32264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:30.892591 32264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:30.894349 32264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:30.894644 32264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:30.895009 32264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:30.901081 32270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:30.901082 32271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:30.901105 32264 server_base.cc:1034] running on GCE node
W20260811 11:15:30.901081 32273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:30.901612 32264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:30.901921 32264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:30.903064 32264 hybrid_clock.cc:648] HybridClock initialized: now 1786446930903011 us; error 62 us; skew 500 ppm
Aug 11 11:15:30 dist-test-slave-2w3w krb5kdc[32142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446930, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:30.906515 32264 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:15:30.907195 32264 webserver.cc:460] Webserver started at http://127.28.25.1:37499/ using document root <none> and password file <none>
I20260811 11:15:30.907460 32264 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:30.907536 32264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:30.907693 32264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:30.908826 32264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "92420ccc578646d5abea9496061b352b"
format_stamp: "Formatted at 2026-08-11 11:15:30 on dist-test-slave-2w3w"
server_key: "34ec7b9cfbe8f6e2012548fdd295fac2"
server_key_iv: "dbb26333d55434dce60fe48b51697c23"
server_key_version: "encryptionkey@0"
I20260811 11:15:30.909211 32264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "92420ccc578646d5abea9496061b352b"
format_stamp: "Formatted at 2026-08-11 11:15:30 on dist-test-slave-2w3w"
server_key: "34ec7b9cfbe8f6e2012548fdd295fac2"
server_key_iv: "dbb26333d55434dce60fe48b51697c23"
server_key_version: "encryptionkey@0"
I20260811 11:15:30.910698 32264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:15:30.911893 32281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:30.912177 32264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:30.912359 32264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "92420ccc578646d5abea9496061b352b"
format_stamp: "Formatted at 2026-08-11 11:15:30 on dist-test-slave-2w3w"
server_key: "34ec7b9cfbe8f6e2012548fdd295fac2"
server_key_iv: "dbb26333d55434dce60fe48b51697c23"
server_key_version: "encryptionkey@0"
I20260811 11:15:30.912452 32264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:30.926935 32264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:30.929919 32264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:30.930128 32264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:30.930416 32264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:30.930765 32264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:30.930822 32264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:30.930878 32264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:30.930907 32264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:30.938046 32264 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:34769
I20260811 11:15:30.938196 32406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:34769 every 8 connection(s)
I20260811 11:15:30.938519 32264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:30.939070 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 32264
I20260811 11:15:30.939181 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:30.939411 28772 external_mini_cluster.cc:1428] Setting key 1ec651b6d1c2dcc82b0f62d7f8bfd0e8
Aug 11 11:15:30 dist-test-slave-2w3w krb5kdc[32142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446930, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
I20260811 11:15:30.950208 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:30.939471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:40595 (local address 127.28.25.62:40767)
0811 11:15:30.939532 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:30.939534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:30.940030 (+ 496us) server_negotiation.cc:407] Connection header received
0811 11:15:30.940375 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:30.940376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:30.940387 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:30.940444 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:30.941883 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:30.942725 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:30.944007 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:30.944241 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:30.944321 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:30.946798 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:30.946800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:30.946800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:30.946809 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:30.948596 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:30.949104 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:30.949105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:30.949106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:30.949169 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:30.949462 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:30.949463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:30.949463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:30.949600 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:30.949673 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:30.949991 (+ 318us) server_negotiation.cc:299] Negotiation successful
0811 11:15:30.950098 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260811 11:15:30.950469 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:30.939441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:40767 (local address 127.28.25.1:40595)
0811 11:15:30.939630 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:30.939676 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:30.940236 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:30.940526 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:30.940531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:30.941445 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:30.941755 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:30.941762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:30.942944 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:30.942945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:30.943895 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:30.943898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:30.944349 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:30.944349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:30.944489 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:30.945150 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:30.945161 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:30.946680 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:30.948730 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:30.948742 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:30.948745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:30.949014 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:30.949255 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:30.949255 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:30.949256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:30.949381 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:30.949677 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:30.949686 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:30.949846 (+ 160us) client_negotiation.cc:769] Sending connection context
0811 11:15:30.949971 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 11:15:30.950181 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260811 11:15:30.951337 32407 heartbeater.cc:347] Connected to a master server at 127.28.25.62:40767
I20260811 11:15:30.951445 32407 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:30.951632 32407 heartbeater.cc:511] Master 127.28.25.62:40767 requested a full tablet report, sending...
I20260811 11:15:30.952332 32196 ts_manager.cc:194] Registered new tserver with Master: 92420ccc578646d5abea9496061b352b (127.28.25.1:34769)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
I20260811 11:15:30.953065 32196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:40595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:15:30.975190 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:40767
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:31.068887 32413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:31.069178 32413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:31.069226 32413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:31.070488 32413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:31.070564 32413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:31.070609 32413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:31.072278 32413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:31.072585 32413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:31.072847 32413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:31.078600 32421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:31.078547 32418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:31.078547 32419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:31.078884 32413 server_base.cc:1034] running on GCE node
I20260811 11:15:31.079250 32413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:31.079578 32413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:31.080757 32413 hybrid_clock.cc:648] HybridClock initialized: now 1786446931080743 us; error 33 us; skew 500 ppm
Aug 11 11:15:31 dist-test-slave-2w3w krb5kdc[32142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446931, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:31.083864 32413 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:15:31.084465 32413 webserver.cc:460] Webserver started at http://127.28.25.2:38769/ using document root <none> and password file <none>
I20260811 11:15:31.084719 32413 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:31.084793 32413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:31.084977 32413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:31.086151 32413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "aa30219952a849c8963e8b65b88cb31f"
format_stamp: "Formatted at 2026-08-11 11:15:31 on dist-test-slave-2w3w"
server_key: "55a323bf7d6264d11c3f6efd5323f82d"
server_key_iv: "6f6f992c56ecd2328258ab3ce4ca869b"
server_key_version: "encryptionkey@0"
I20260811 11:15:31.086565 32413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "aa30219952a849c8963e8b65b88cb31f"
format_stamp: "Formatted at 2026-08-11 11:15:31 on dist-test-slave-2w3w"
server_key: "55a323bf7d6264d11c3f6efd5323f82d"
server_key_iv: "6f6f992c56ecd2328258ab3ce4ca869b"
server_key_version: "encryptionkey@0"
I20260811 11:15:31.088186 32413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:15:31.089151 32429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:31.089375 32413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:31.089496 32413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "aa30219952a849c8963e8b65b88cb31f"
format_stamp: "Formatted at 2026-08-11 11:15:31 on dist-test-slave-2w3w"
server_key: "55a323bf7d6264d11c3f6efd5323f82d"
server_key_iv: "6f6f992c56ecd2328258ab3ce4ca869b"
server_key_version: "encryptionkey@0"
I20260811 11:15:31.089581 32413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:31.106037 32413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:31.108764 32413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:31.108963 32413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:31.109229 32413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:31.109565 32413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:31.109620 32413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:31.109663 32413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:31.109700 32413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:31.116605 32413 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:39581
I20260811 11:15:31.116726 32558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:39581 every 8 connection(s)
I20260811 11:15:31.117040 32413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:15:31.121986 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 32413
I20260811 11:15:31.122120 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:31.122303 28772 external_mini_cluster.cc:1428] Setting key 7f89099557484efb361544d77909d207
Aug 11 11:15:31 dist-test-slave-2w3w krb5kdc[32142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446931, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:31.129120 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:31.117833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:41387 (local address 127.28.25.62:40767)
0811 11:15:31.117926 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:31.117929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:31.118103 (+ 174us) server_negotiation.cc:407] Connection header received
0811 11:15:31.118620 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:31.118621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:31.118634 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:31.118694 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:31.119807 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:31.120304 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:31.121342 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:31.121537 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:31.121607 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:31.125120 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:31.125122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:31.125123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:31.125134 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:31.127038 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:31.127636 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:31.127638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:31.127638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:31.127710 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:31.128084 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:31.128085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:31.128085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:31.128290 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:31.128374 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:31.128772 (+ 398us) server_negotiation.cc:299] Negotiation successful
0811 11:15:31.128952 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 11:15:31.129173 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:31.117783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:40767 (local address 127.28.25.2:41387)
0811 11:15:31.117963 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:31.118004 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:31.118498 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:31.118756 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:31.118759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:31.119474 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:31.119694 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:31.119699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:31.120399 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:31.120399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:31.121234 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:31.121236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:31.121638 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:31.121638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:31.121852 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:31.122928 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:31.122938 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:31.124995 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:31.127134 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:31.127147 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:31.127150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:31.127538 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:31.127797 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:31.127799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:31.127799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:31.127995 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:31.128385 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:31.128396 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:31.128612 (+ 216us) client_negotiation.cc:769] Sending connection context
0811 11:15:31.128746 (+ 134us) client_negotiation.cc:240] Negotiation successful
0811 11:15:31.128954 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260811 11:15:31.129840 32559 heartbeater.cc:347] Connected to a master server at 127.28.25.62:40767
I20260811 11:15:31.129963 32559 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:31.130169 32559 heartbeater.cc:511] Master 127.28.25.62:40767 requested a full tablet report, sending...
I20260811 11:15:31.130673 32196 ts_manager.cc:194] Registered new tserver with Master: aa30219952a849c8963e8b65b88cb31f (127.28.25.2:39581)
I20260811 11:15:31.131313 32196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:41387
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:15:31.160822 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:40767
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:31.248689 32566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:31.248921 32566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:31.248948 32566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:31.250310 32566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:31.250360 32566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:31.250388 32566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:31.252005 32566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:31.252283 32566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:31.252543 32566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:31.258255 32577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:31.258361 32566 server_base.cc:1034] running on GCE node
W20260811 11:15:31.258265 32573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:31.258255 32574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:31.258811 32566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:31.259135 32566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:31.260342 32566 hybrid_clock.cc:648] HybridClock initialized: now 1786446931260335 us; error 41 us; skew 500 ppm
Aug 11 11:15:31 dist-test-slave-2w3w krb5kdc[32142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446931, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:31.263597 32566 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:15:31.264097 32566 webserver.cc:460] Webserver started at http://127.28.25.3:35353/ using document root <none> and password file <none>
I20260811 11:15:31.264341 32566 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:31.264392 32566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:31.264501 32566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:31.265677 32566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "99f4163a446042c5a06982ed0ac97876"
format_stamp: "Formatted at 2026-08-11 11:15:31 on dist-test-slave-2w3w"
server_key: "b3584edfa946e7aaa4a7e3576626aa64"
server_key_iv: "25eb3ead24aaf132af9d80b5b0d81e37"
server_key_version: "encryptionkey@0"
I20260811 11:15:31.266045 32566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "99f4163a446042c5a06982ed0ac97876"
format_stamp: "Formatted at 2026-08-11 11:15:31 on dist-test-slave-2w3w"
server_key: "b3584edfa946e7aaa4a7e3576626aa64"
server_key_iv: "25eb3ead24aaf132af9d80b5b0d81e37"
server_key_version: "encryptionkey@0"
I20260811 11:15:31.268069 32566 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 11:15:31.269023 32586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:31.269296 32566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:15:31.269380 32566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "99f4163a446042c5a06982ed0ac97876"
format_stamp: "Formatted at 2026-08-11 11:15:31 on dist-test-slave-2w3w"
server_key: "b3584edfa946e7aaa4a7e3576626aa64"
server_key_iv: "25eb3ead24aaf132af9d80b5b0d81e37"
server_key_version: "encryptionkey@0"
I20260811 11:15:31.269447 32566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:31.287712 32566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:31.289932 32566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:31.290102 32566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:31.290316 32566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:31.290696 32566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:31.290735 32566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:31.290788 32566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:31.290817 32566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:31.297772 32566 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:46627
I20260811 11:15:31.297886 32713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:46627 every 8 connection(s)
I20260811 11:15:31.298231 32566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
Aug 11 11:15:31 dist-test-slave-2w3w krb5kdc[32142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446931, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:31.307633 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 32566
I20260811 11:15:31.307765 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:15:31.308019 28772 external_mini_cluster.cc:1428] Setting key 997264f5836ccd808e8dc97d4c0c804e
I20260811 11:15:31.308961 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:31.299198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:48995 (local address 127.28.25.62:40767)
0811 11:15:31.299302 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:31.299305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:31.299473 (+ 168us) server_negotiation.cc:407] Connection header received
0811 11:15:31.300060 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:31.300061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:31.300074 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:31.300139 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:31.301225 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:31.301749 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:31.302835 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:31.303148 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:31.303230 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:31.305596 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:31.305597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:31.305598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:31.305606 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:31.307154 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:31.307676 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:31.307677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:31.307678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:31.307744 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:31.308028 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:31.308029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:31.308029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:31.308182 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:31.308272 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:31.308759 (+ 487us) server_negotiation.cc:299] Negotiation successful
0811 11:15:31.308860 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 11:15:31.309320 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:31.299111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:40767 (local address 127.28.25.3:48995)
0811 11:15:31.299329 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:31.299378 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:31.299937 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:31.300218 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:31.300221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:31.300838 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:31.301110 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:31.301115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:31.301866 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:31.301867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:31.302726 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:31.302729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:31.303247 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:31.303248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:31.303400 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:31.303980 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:31.303989 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:31.305462 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:31.307290 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:31.307301 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:31.307304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:31.307578 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:31.307828 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:31.307829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:31.307830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:31.307955 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:31.308296 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:31.308306 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:31.308547 (+ 241us) client_negotiation.cc:769] Sending connection context
0811 11:15:31.308716 (+ 169us) client_negotiation.cc:240] Negotiation successful
0811 11:15:31.309000 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":98,"threads_started":1}
I20260811 11:15:31.309931 32714 heartbeater.cc:347] Connected to a master server at 127.28.25.62:40767
I20260811 11:15:31.310029 32714 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:31.310267 32714 heartbeater.cc:511] Master 127.28.25.62:40767 requested a full tablet report, sending...
I20260811 11:15:31.310701 32196 ts_manager.cc:194] Registered new tserver with Master: 99f4163a446042c5a06982ed0ac97876 (127.28.25.3:46627)
I20260811 11:15:31.311252 32196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:48995
I20260811 11:15:31.320474 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 11:15:31 dist-test-slave-2w3w krb5kdc[32142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446931, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 11 11:15:31 dist-test-slave-2w3w krb5kdc[32142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446931, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:31.346697 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:31.338541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.28.25.62:40767)
0811 11:15:31.338616 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:31.338618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:31.338721 (+ 103us) server_negotiation.cc:407] Connection header received
0811 11:15:31.338757 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:31.338758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:31.338769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:31.338841 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:31.339622 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:31.340132 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:31.340975 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:31.341160 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:31.341216 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:31.343785 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:31.343786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:31.343786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:31.343795 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:31.345365 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:31.345731 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:31.345732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:31.345732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:31.345791 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:31.346100 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:31.346101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:31.346101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:31.346295 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:31.346404 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:31.346532 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 11:15:31.346621 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260811 11:15:31.347066 32196 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59242
I20260811 11:15:31.347638 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 32264
I20260811 11:15:31.355134 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 32413
I20260811 11:15:31.361770 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 32566
I20260811 11:15:31.369606 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 32158
2026-08-11T11:15:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2044 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:31 dist-test-slave-2w3w krb5kdc[32727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:31 dist-test-slave-2w3w krb5kdc[32727](info): set up 2 sockets
Aug 11 11:15:31 dist-test-slave-2w3w krb5kdc[32727](info): commencing operation
krb5kdc: starting...
W20260811 11:15:32.501200 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:32 dist-test-slave-2w3w krb5kdc[32727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:15:32.643193 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:41241
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:41241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:32.737123 32743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:32.737440 32743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:32.737490 32743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:32.738814 32743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:32.738909 32743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:32.738933 32743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:32.738956 32743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:32.738974 32743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:32.740811 32743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:41241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:41241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:32.741154 32743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:32.741559 32743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:15:32.747171 32743 server_base.cc:1034] running on GCE node
W20260811 11:15:32.747188 32751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:32.747206 32749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:32.747332 32748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:32.747618 32743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:32.747905 32743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:32.749089 32743 hybrid_clock.cc:648] HybridClock initialized: now 1786446932749058 us; error 38 us; skew 500 ppm
Aug 11 11:15:32 dist-test-slave-2w3w krb5kdc[32727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446932, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:32.752269 32743 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:15:32.752802 32743 webserver.cc:460] Webserver started at http://127.28.25.62:43553/ using document root <none> and password file <none>
I20260811 11:15:32.753059 32743 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:32.753134 32743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:32.753297 32743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:32.754472 32743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "e505736f30424aeda22d99f3aa5310c3"
format_stamp: "Formatted at 2026-08-11 11:15:32 on dist-test-slave-2w3w"
server_key: "2c7b1706c4ce9e013be8289d13d7e870"
server_key_iv: "86908d7a444966a539f1397534ca73a4"
server_key_version: "encryptionkey@0"
I20260811 11:15:32.754886 32743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "e505736f30424aeda22d99f3aa5310c3"
format_stamp: "Formatted at 2026-08-11 11:15:32 on dist-test-slave-2w3w"
server_key: "2c7b1706c4ce9e013be8289d13d7e870"
server_key_iv: "86908d7a444966a539f1397534ca73a4"
server_key_version: "encryptionkey@0"
I20260811 11:15:32.756453 32743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:32.757380 32758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:32.757627 32743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:32.757746 32743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "e505736f30424aeda22d99f3aa5310c3"
format_stamp: "Formatted at 2026-08-11 11:15:32 on dist-test-slave-2w3w"
server_key: "2c7b1706c4ce9e013be8289d13d7e870"
server_key_iv: "86908d7a444966a539f1397534ca73a4"
server_key_version: "encryptionkey@0"
I20260811 11:15:32.757834 32743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:32.773041 32743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:32.775853 32743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:32.776064 32743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:32.780843 32743 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:41241
I20260811 11:15:32.780884 351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:41241 every 8 connection(s)
I20260811 11:15:32.781530 32743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:32.782351 352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:32.785053 352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3: Bootstrap starting.
I20260811 11:15:32.785765 352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:32.786108 352 log.cc:824] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:32.787056 352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3: No bootstrap required, opened a new log
I20260811 11:15:32.788897 352 raft_consensus.cc:348] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e505736f30424aeda22d99f3aa5310c3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41241 } }
I20260811 11:15:32.789060 352 raft_consensus.cc:374] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:32.789083 352 raft_consensus.cc:729] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e505736f30424aeda22d99f3aa5310c3, State: Initialized, Role: FOLLOWER
I20260811 11:15:32.789255 352 consensus_queue.cc:260] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e505736f30424aeda22d99f3aa5310c3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41241 } }
I20260811 11:15:32.789347 352 raft_consensus.cc:388] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:32.789391 352 raft_consensus.cc:482] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:32.789448 352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:32.790429 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 32743
I20260811 11:15:32.790529 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:32.790534 352 raft_consensus.cc:504] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e505736f30424aeda22d99f3aa5310c3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41241 } }
I20260811 11:15:32.790689 352 leader_election.cc:302] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e505736f30424aeda22d99f3aa5310c3; no voters:
I20260811 11:15:32.790746 28772 external_mini_cluster.cc:1428] Setting key 06513d2ceee4b42b11c202b739fdc25a
I20260811 11:15:32.790913 352 leader_election.cc:288] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:32.790979 356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:32.791196 356 raft_consensus.cc:686] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [term 1 LEADER]: Becoming Leader. State: Replica: e505736f30424aeda22d99f3aa5310c3, State: Running, Role: LEADER
I20260811 11:15:32.791381 352 sys_catalog.cc:564] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:32.791392 356 consensus_queue.cc:237] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e505736f30424aeda22d99f3aa5310c3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41241 } }
I20260811 11:15:32.791854 357 sys_catalog.cc:455] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e505736f30424aeda22d99f3aa5310c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e505736f30424aeda22d99f3aa5310c3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41241 } } }
I20260811 11:15:32.791860 359 sys_catalog.cc:455] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e505736f30424aeda22d99f3aa5310c3. Latest consensus state: current_term: 1 leader_uuid: "e505736f30424aeda22d99f3aa5310c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e505736f30424aeda22d99f3aa5310c3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41241 } } }
I20260811 11:15:32.791966 357 sys_catalog.cc:458] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:32.791996 359 sys_catalog.cc:458] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:32.792238 366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:32.792922 366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:32.795077 366 catalog_manager.cc:1347] Generated new cluster ID: 565738e1ce1248289e99eaf547ef0cd9
I20260811 11:15:32.795167 366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 11:15:32 dist-test-slave-2w3w krb5kdc[32727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:32.805516 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:32.791404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.28.25.62:41241)
0811 11:15:32.791581 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:32.791593 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:32.791912 (+ 319us) server_negotiation.cc:407] Connection header received
0811 11:15:32.792056 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:32.792065 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:32.792138 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:32.792292 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:32.793407 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:32.794751 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:32.795892 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:32.796625 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:32.796743 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:32.799997 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:32.800003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:32.800003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:32.800020 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:32.803046 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:32.803524 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:32.803528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:32.803529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:32.803662 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:32.803929 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:32.803929 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:32.803930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:32.804310 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:32.804504 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:32.805102 (+ 598us) server_negotiation.cc:299] Negotiation successful
0811 11:15:32.805269 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260811 11:15:32.811335 366 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:32.811972 366 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:32.820525 366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e505736f30424aeda22d99f3aa5310c3: Generated new TSK 0
I20260811 11:15:32.820796 366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:15:32.858834 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41241
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:32.949709 383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:32.949968 383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:32.949997 383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:32.951406 383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:32.951462 383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:32.951494 383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:32.953167 383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:32.953400 383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:32.953614 383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:32.959507 389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:32.959507 388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:32.959513 391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:32.959975 383 server_base.cc:1034] running on GCE node
I20260811 11:15:32.960229 383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:32.960558 383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:32.961735 383 hybrid_clock.cc:648] HybridClock initialized: now 1786446932961703 us; error 53 us; skew 500 ppm
Aug 11 11:15:32 dist-test-slave-2w3w krb5kdc[32727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446932, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:32.965164 383 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:15:32.965724 383 webserver.cc:460] Webserver started at http://127.28.25.1:44727/ using document root <none> and password file <none>
I20260811 11:15:32.966010 383 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:32.966068 383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:32.966213 383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:32.967504 383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "5b221058d6ea40ceadf73a8884e15f50"
format_stamp: "Formatted at 2026-08-11 11:15:32 on dist-test-slave-2w3w"
server_key: "60620166ed9e5427c8ae693a8fa74f43"
server_key_iv: "4dd1a0bb72200794c9e2940e07bd3c06"
server_key_version: "encryptionkey@0"
I20260811 11:15:32.967895 383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "5b221058d6ea40ceadf73a8884e15f50"
format_stamp: "Formatted at 2026-08-11 11:15:32 on dist-test-slave-2w3w"
server_key: "60620166ed9e5427c8ae693a8fa74f43"
server_key_iv: "4dd1a0bb72200794c9e2940e07bd3c06"
server_key_version: "encryptionkey@0"
I20260811 11:15:32.969427 383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260811 11:15:32.970306 400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:32.970548 383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:32.970656 383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "5b221058d6ea40ceadf73a8884e15f50"
format_stamp: "Formatted at 2026-08-11 11:15:32 on dist-test-slave-2w3w"
server_key: "60620166ed9e5427c8ae693a8fa74f43"
server_key_iv: "4dd1a0bb72200794c9e2940e07bd3c06"
server_key_version: "encryptionkey@0"
I20260811 11:15:32.970726 383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:32.987763 383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:32.990584 383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:32.990724 383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:32.991079 383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:32.991614 383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:32.991657 383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:32.991681 383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:32.991694 383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:32.998371 383 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:40997
I20260811 11:15:32.998485 529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:40997 every 8 connection(s)
I20260811 11:15:32.998854 383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:33.005769 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 383
I20260811 11:15:33.005924 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:33.006136 28772 external_mini_cluster.cc:1428] Setting key 4a482b4cc7b47e0de2844310a58d6569
Aug 11 11:15:32 dist-test-slave-2w3w krb5kdc[32727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446932, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:33.010258 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:32.999848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:49713 (local address 127.28.25.62:41241)
0811 11:15:32.999922 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:32.999924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:33.000154 (+ 230us) server_negotiation.cc:407] Connection header received
0811 11:15:33.001158 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:33.001160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:33.001178 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:33.001294 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:33.002039 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.002599 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.003870 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.004078 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.004142 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:33.006920 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:33.006922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:33.006922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:33.006931 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:33.008614 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:33.009130 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:33.009132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:33.009132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:33.009202 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:33.009507 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:33.009508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:33.009508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:33.009639 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:33.009722 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:33.010038 (+ 316us) server_negotiation.cc:299] Negotiation successful
0811 11:15:33.010157 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:15:33.010489 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:32.999759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:41241 (local address 127.28.25.1:49713)
0811 11:15:32.999972 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:33.000019 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:33.001014 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:33.001364 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:33.001368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:33.001658 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:33.001900 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:33.001906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.002714 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.002716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:33.003752 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:33.003755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.004167 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.004168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:33.004285 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:33.004959 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:33.004969 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:33.006749 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:33.008737 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:33.008749 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:33.008752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:33.009036 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:33.009279 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:33.009280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:33.009280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:33.009422 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:33.009711 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:33.009720 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:33.009891 (+ 171us) client_negotiation.cc:769] Sending connection context
0811 11:15:33.010014 (+ 123us) client_negotiation.cc:240] Negotiation successful
0811 11:15:33.010203 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":124,"threads_started":1}
I20260811 11:15:33.011493 530 heartbeater.cc:347] Connected to a master server at 127.28.25.62:41241
I20260811 11:15:33.011648 530 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:33.011955 530 heartbeater.cc:511] Master 127.28.25.62:41241 requested a full tablet report, sending...
I20260811 11:15:33.012648 310 ts_manager.cc:194] Registered new tserver with Master: 5b221058d6ea40ceadf73a8884e15f50 (127.28.25.1:40997)
I20260811 11:15:33.013410 310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:49713
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:15:33.039944 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41241
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:33.133316 536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:33.133576 536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:33.133607 536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:33.135032 536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:33.135095 536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:33.135128 536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:33.137013 536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:33.137291 536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:33.137501 536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:33.143518 542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:33.143541 544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:33.143564 541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:33.144021 536 server_base.cc:1034] running on GCE node
I20260811 11:15:33.144302 536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:33.144594 536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:33.145759 536 hybrid_clock.cc:648] HybridClock initialized: now 1786446933145744 us; error 41 us; skew 500 ppm
Aug 11 11:15:33 dist-test-slave-2w3w krb5kdc[32727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446933, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:33.148660 536 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:15:33.149201 536 webserver.cc:460] Webserver started at http://127.28.25.2:40829/ using document root <none> and password file <none>
I20260811 11:15:33.149479 536 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:33.149555 536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:33.149717 536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:33.150875 536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "1026f15f136846a6a010d053781471b0"
format_stamp: "Formatted at 2026-08-11 11:15:33 on dist-test-slave-2w3w"
server_key: "3e7096f7ccd84f662948c66225e503f0"
server_key_iv: "cb95e75b2d0ca572126cfe8ad1c6f732"
server_key_version: "encryptionkey@0"
I20260811 11:15:33.151361 536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "1026f15f136846a6a010d053781471b0"
format_stamp: "Formatted at 2026-08-11 11:15:33 on dist-test-slave-2w3w"
server_key: "3e7096f7ccd84f662948c66225e503f0"
server_key_iv: "cb95e75b2d0ca572126cfe8ad1c6f732"
server_key_version: "encryptionkey@0"
I20260811 11:15:33.153041 536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:15:33.154112 551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:33.154402 536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:33.154524 536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "1026f15f136846a6a010d053781471b0"
format_stamp: "Formatted at 2026-08-11 11:15:33 on dist-test-slave-2w3w"
server_key: "3e7096f7ccd84f662948c66225e503f0"
server_key_iv: "cb95e75b2d0ca572126cfe8ad1c6f732"
server_key_version: "encryptionkey@0"
I20260811 11:15:33.154618 536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:33.165264 536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:33.167860 536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:33.168056 536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:33.168332 536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:33.168709 536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:33.168766 536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:33.168808 536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:33.168853 536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:33.175994 536 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:44861
I20260811 11:15:33.176060 687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:44861 every 8 connection(s)
I20260811 11:15:33.176447 536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:15:33.176725 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 536
I20260811 11:15:33.176898 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:33.177143 28772 external_mini_cluster.cc:1428] Setting key 145abcdde6f2654c0362ec480fcf29da
Aug 11 11:15:33 dist-test-slave-2w3w krb5kdc[32727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446933, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
I20260811 11:15:33.188464 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:33.177284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:53115 (local address 127.28.25.62:41241)
0811 11:15:33.177349 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:33.177352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:33.177706 (+ 354us) server_negotiation.cc:407] Connection header received
0811 11:15:33.179068 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:33.179069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:33.179084 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:33.179134 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:33.179936 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.180506 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.181856 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.182083 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.182147 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:33.185032 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:33.185034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:33.185034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:33.185043 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:33.186659 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:33.187285 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:33.187287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:33.187287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:33.187356 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:33.187717 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:33.187718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:33.187718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:33.187857 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:33.187939 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:33.188241 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 11:15:33.188357 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 11:15:33.188659 690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:33.177288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:41241 (local address 127.28.25.2:53115)
0811 11:15:33.177516 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:33.177581 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:33.178880 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:33.179209 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:33.179213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:33.179525 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:33.179815 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:33.179823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.180624 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.180625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:33.181759 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:33.181762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.182187 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.182189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:33.182367 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:33.182992 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:33.183004 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:33.184908 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:33.186768 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:33.186779 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:33.186782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:33.187197 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:33.187461 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:33.187462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:33.187462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:33.187632 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:33.187928 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:33.187936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:33.188095 (+ 159us) client_negotiation.cc:769] Sending connection context
0811 11:15:33.188205 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 11:15:33.188396 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260811 11:15:33.189411 688 heartbeater.cc:347] Connected to a master server at 127.28.25.62:41241
I20260811 11:15:33.189524 688 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:33.189808 688 heartbeater.cc:511] Master 127.28.25.62:41241 requested a full tablet report, sending...
I20260811 11:15:33.190335 310 ts_manager.cc:194] Registered new tserver with Master: 1026f15f136846a6a010d053781471b0 (127.28.25.2:44861)
I20260811 11:15:33.190724 310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:53115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:15:33.211565 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41241
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:33.299736 693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:33.300027 693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:33.300076 693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:33.301425 693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:33.301510 693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:33.301563 693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:33.303406 693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:33.303722 693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:33.303988 693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:33.309832 703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:33.309865 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:33.309840 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:33.309885 693 server_base.cc:1034] running on GCE node
I20260811 11:15:33.310281 693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:33.310518 693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:33.311688 693 hybrid_clock.cc:648] HybridClock initialized: now 1786446933311677 us; error 34 us; skew 500 ppm
Aug 11 11:15:33 dist-test-slave-2w3w krb5kdc[32727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446933, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:33.314790 693 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:15:33.315469 693 webserver.cc:460] Webserver started at http://127.28.25.3:43115/ using document root <none> and password file <none>
I20260811 11:15:33.315729 693 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:33.315783 693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:33.315963 693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:33.317144 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "e1311b2ee77747f2833c546e5b79c32b"
format_stamp: "Formatted at 2026-08-11 11:15:33 on dist-test-slave-2w3w"
server_key: "56b97641a4009c8c5d88901f0535a83a"
server_key_iv: "bc0470840ac07c5344dd0344ff335fe5"
server_key_version: "encryptionkey@0"
I20260811 11:15:33.317553 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "e1311b2ee77747f2833c546e5b79c32b"
format_stamp: "Formatted at 2026-08-11 11:15:33 on dist-test-slave-2w3w"
server_key: "56b97641a4009c8c5d88901f0535a83a"
server_key_iv: "bc0470840ac07c5344dd0344ff335fe5"
server_key_version: "encryptionkey@0"
I20260811 11:15:33.319169 693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:15:33.320106 710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:33.320324 693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:33.320459 693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "e1311b2ee77747f2833c546e5b79c32b"
format_stamp: "Formatted at 2026-08-11 11:15:33 on dist-test-slave-2w3w"
server_key: "56b97641a4009c8c5d88901f0535a83a"
server_key_iv: "bc0470840ac07c5344dd0344ff335fe5"
server_key_version: "encryptionkey@0"
I20260811 11:15:33.320552 693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:33.339998 693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:33.342442 693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:33.342619 693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:33.343025 693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:33.343421 693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:33.343482 693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:33.343533 693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:33.343560 693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:33.350044 693 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:33581
I20260811 11:15:33.350091 842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:33581 every 8 connection(s)
I20260811 11:15:33.350507 693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
Aug 11 11:15:33 dist-test-slave-2w3w krb5kdc[32727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446933, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:33.358431 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 693
I20260811 11:15:33.358538 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:15:33.358801 28772 external_mini_cluster.cc:1428] Setting key 7c935c6b8e2ab6a677a2ba352f1f8210
I20260811 11:15:33.362315 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:33.351457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:46797 (local address 127.28.25.62:41241)
0811 11:15:33.351543 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:33.351546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:33.351775 (+ 229us) server_negotiation.cc:407] Connection header received
0811 11:15:33.352986 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:33.352988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:33.353000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:33.353066 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:33.353844 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.354359 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.355588 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.355841 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.355907 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:33.358527 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:33.358530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:33.358530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:33.358541 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:33.360638 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:33.361203 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:33.361204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:33.361204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:33.361284 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:33.361543 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:33.361543 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:33.361543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:33.361715 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:33.361834 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:33.362154 (+ 320us) server_negotiation.cc:299] Negotiation successful
0811 11:15:33.362242 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:15:33.362555 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:33.351406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:41241 (local address 127.28.25.3:46797)
0811 11:15:33.351626 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:33.351677 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:33.352794 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:33.353166 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:33.353170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:33.353458 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:33.353676 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:33.353680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.354470 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.354471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:33.355486 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:33.355489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.355927 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.355928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:33.356038 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:33.356655 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:33.356664 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:33.358360 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:33.360729 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:33.360738 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:33.360740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:33.361094 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:33.361359 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:33.361359 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:33.361359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:33.361481 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:33.361859 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:33.361866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:33.362018 (+ 152us) client_negotiation.cc:769] Sending connection context
0811 11:15:33.362123 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 11:15:33.362276 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":107,"threads_started":1}
I20260811 11:15:33.363188 843 heartbeater.cc:347] Connected to a master server at 127.28.25.62:41241
I20260811 11:15:33.363303 843 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:33.363474 843 heartbeater.cc:511] Master 127.28.25.62:41241 requested a full tablet report, sending...
I20260811 11:15:33.363870 310 ts_manager.cc:194] Registered new tserver with Master: e1311b2ee77747f2833c546e5b79c32b (127.28.25.3:33581)
I20260811 11:15:33.364223 310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:46797
I20260811 11:15:33.371438 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 11:15:33 dist-test-slave-2w3w krb5kdc[32727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 11:15:33 dist-test-slave-2w3w krb5kdc[32727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
I20260811 11:15:33.400839 850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:33.391626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.28.25.1:40997)
0811 11:15:33.391811 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:33.391822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:33.391834 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:15:33.391864 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:33.391866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:33.391882 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:33.391979 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:33.392870 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.393632 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.394464 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.394873 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.394952 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:33.397542 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:33.397555 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:33.397555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:33.397563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:33.399446 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:33.399874 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:33.399878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:33.399878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:33.399966 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:33.400165 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:33.400165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:33.400166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:33.400332 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:33.400461 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:33.400660 (+ 199us) server_negotiation.cc:299] Negotiation successful
0811 11:15:33.400740 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
Aug 11 11:15:33 dist-test-slave-2w3w krb5kdc[32727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:33.409777 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:33.402477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.28.25.62:41241)
0811 11:15:33.402535 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:33.402539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:33.402552 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:15:33.402684 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:33.402685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:33.402697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:33.402760 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:33.403490 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.404031 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.404724 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:33.404917 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:33.404966 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:33.407419 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:33.407421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:33.407421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:33.407428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:33.408654 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:33.409009 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:33.409009 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:33.409010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:33.409063 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:33.409224 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:33.409224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:33.409225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:33.409396 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:33.409477 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:33.409638 (+ 161us) server_negotiation.cc:299] Negotiation successful
0811 11:15:33.409708 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260811 11:15:33.410132 310 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37382
I20260811 11:15:33.410547 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 383
I20260811 11:15:33.420022 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 536
I20260811 11:15:33.429589 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 693
I20260811 11:15:33.438943 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 32743
2026-08-11T11:15:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2069 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:33 dist-test-slave-2w3w krb5kdc[855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:33 dist-test-slave-2w3w krb5kdc[855](info): set up 2 sockets
Aug 11 11:15:33 dist-test-slave-2w3w krb5kdc[855](info): commencing operation
krb5kdc: starting...
W20260811 11:15:34.585129 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:34 dist-test-slave-2w3w krb5kdc[855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:15:34.722625 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:38619
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:34.810921 868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:34.811235 868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:34.811282 868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:34.812566 868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:34.812652 868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:34.812675 868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:34.812695 868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:34.812742 868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:34.814412 868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:38619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:38619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:34.814743 868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:34.815209 868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:34.820469 875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:34.820451 877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:34.820456 874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:34.820737 868 server_base.cc:1034] running on GCE node
I20260811 11:15:34.821034 868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:34.821374 868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:34.822554 868 hybrid_clock.cc:648] HybridClock initialized: now 1786446934822541 us; error 50 us; skew 500 ppm
Aug 11 11:15:34 dist-test-slave-2w3w krb5kdc[855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446934, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:34.825403 868 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:15:34.825829 868 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:34.825882 868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:34.825980 868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:34.827160 868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "4333a5b1ce8940688ae5ce7079db0e05"
format_stamp: "Formatted at 2026-08-11 11:15:34 on dist-test-slave-2w3w"
server_key: "5824af9f0a95aa18855b81fd63021116"
server_key_iv: "e854cbe802fe11c63835a2b5b10c2bf1"
server_key_version: "encryptionkey@0"
I20260811 11:15:34.827483 868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "4333a5b1ce8940688ae5ce7079db0e05"
format_stamp: "Formatted at 2026-08-11 11:15:34 on dist-test-slave-2w3w"
server_key: "5824af9f0a95aa18855b81fd63021116"
server_key_iv: "e854cbe802fe11c63835a2b5b10c2bf1"
server_key_version: "encryptionkey@0"
I20260811 11:15:34.828900 868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 11:15:34.829790 883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:34.830052 868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:34.830149 868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "4333a5b1ce8940688ae5ce7079db0e05"
format_stamp: "Formatted at 2026-08-11 11:15:34 on dist-test-slave-2w3w"
server_key: "5824af9f0a95aa18855b81fd63021116"
server_key_iv: "e854cbe802fe11c63835a2b5b10c2bf1"
server_key_version: "encryptionkey@0"
I20260811 11:15:34.830215 868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:34.845435 868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:34.848174 868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:34.848309 868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:34.852648 868 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:38619
I20260811 11:15:34.852677 943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:38619 every 8 connection(s)
I20260811 11:15:34.853034 868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:34.854212 944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:34.856786 944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05: Bootstrap starting.
I20260811 11:15:34.857410 944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:34.857681 944 log.cc:824] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:34.858424 944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05: No bootstrap required, opened a new log
I20260811 11:15:34.859220 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 868
I20260811 11:15:34.859316 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:34.859541 28772 external_mini_cluster.cc:1428] Setting key 720e85b520bf8032af71abd749283b3c
I20260811 11:15:34.860771 944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333a5b1ce8940688ae5ce7079db0e05" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38619 } }
I20260811 11:15:34.860980 944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:34.861030 944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4333a5b1ce8940688ae5ce7079db0e05, State: Initialized, Role: FOLLOWER
I20260811 11:15:34.861225 944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333a5b1ce8940688ae5ce7079db0e05" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38619 } }
I20260811 11:15:34.861326 944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:34.861364 944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:34.861450 944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:34.862214 944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333a5b1ce8940688ae5ce7079db0e05" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38619 } }
I20260811 11:15:34.862313 944 leader_election.cc:302] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4333a5b1ce8940688ae5ce7079db0e05; no voters:
I20260811 11:15:34.862537 944 leader_election.cc:288] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:34.862579 951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:34.862737 951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [term 1 LEADER]: Becoming Leader. State: Replica: 4333a5b1ce8940688ae5ce7079db0e05, State: Running, Role: LEADER
I20260811 11:15:34.862847 944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:34.862937 951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333a5b1ce8940688ae5ce7079db0e05" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38619 } }
I20260811 11:15:34.863364 951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4333a5b1ce8940688ae5ce7079db0e05. Latest consensus state: current_term: 1 leader_uuid: "4333a5b1ce8940688ae5ce7079db0e05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333a5b1ce8940688ae5ce7079db0e05" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38619 } } }
I20260811 11:15:34.863554 951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:34.864202 961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:34.864787 961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:34.865667 954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4333a5b1ce8940688ae5ce7079db0e05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333a5b1ce8940688ae5ce7079db0e05" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38619 } } }
I20260811 11:15:34.865758 954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05 [sys.catalog]: This master's current role is: LEADER
Aug 11 11:15:34 dist-test-slave-2w3w krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:34.867424 961 catalog_manager.cc:1347] Generated new cluster ID: 937ea8162a0a47bfb32671db1b17e2d9
I20260811 11:15:34.867487 961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:34.873099 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:34.860203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.28.25.62:38619)
0811 11:15:34.860433 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:34.860441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:34.860471 (+ 30us) server_negotiation.cc:407] Connection header received
0811 11:15:34.860736 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:34.860742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:34.860793 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:34.860930 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:34.861778 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:34.863085 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:34.864203 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:34.864910 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:34.865024 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:34.867901 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:34.867906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:34.867906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:34.867919 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:34.870120 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:34.870518 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:34.870522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:34.870522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:34.870656 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:34.870868 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:34.870869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:34.870869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:34.871254 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:34.871428 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:34.871516 (+ 88us) server_negotiation.cc:299] Negotiation successful
0811 11:15:34.872774 (+ 1258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
I20260811 11:15:34.880335 961 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:34.881140 961 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:34.893257 961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05: Generated new TSK 0
I20260811 11:15:34.893532 961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:15:34.926736 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:38619
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:35.016351 977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:35.016636 977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:35.016686 977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:35.018011 977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:35.018101 977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:35.018155 977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:35.019956 977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:38619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:35.020268 977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:35.020535 977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:35.026288 985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:35.026328 987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:35.026335 984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:35.026523 977 server_base.cc:1034] running on GCE node
I20260811 11:15:35.026811 977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:35.027159 977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:35.028349 977 hybrid_clock.cc:648] HybridClock initialized: now 1786446935028327 us; error 39 us; skew 500 ppm
Aug 11 11:15:35 dist-test-slave-2w3w krb5kdc[855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:35.031353 977 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:15:35.031749 977 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:35.031807 977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:35.031914 977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:35.033113 977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "bf49e36cb1034d189e58779fea09d36e"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "363eab12fd6ad63dd9845c4e2bfc8a60"
server_key_iv: "6fbab7bbf1e93949fa24ec3a331a3f15"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.033468 977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "bf49e36cb1034d189e58779fea09d36e"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "363eab12fd6ad63dd9845c4e2bfc8a60"
server_key_iv: "6fbab7bbf1e93949fa24ec3a331a3f15"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.035223 977 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:15:35.036262 994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.036550 977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:35.036660 977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "bf49e36cb1034d189e58779fea09d36e"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "363eab12fd6ad63dd9845c4e2bfc8a60"
server_key_iv: "6fbab7bbf1e93949fa24ec3a331a3f15"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.036756 977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:35.057425 977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:35.060184 977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:35.060365 977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:35.060647 977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:35.060999 977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:35.061055 977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.061100 977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:35.061128 977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.068136 977 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:37897
I20260811 11:15:35.068199 1134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:37897 every 8 connection(s)
I20260811 11:15:35.068486 977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:35.073817 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 977
I20260811 11:15:35.073997 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:35.074237 28772 external_mini_cluster.cc:1428] Setting key 1c148138d740fc17f3ae766401d6a04a
Aug 11 11:15:35 dist-test-slave-2w3w krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:35.079517 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.069304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:42931 (local address 127.28.25.62:38619)
0811 11:15:35.069402 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.069405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.069589 (+ 184us) server_negotiation.cc:407] Connection header received
0811 11:15:35.070136 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.070137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.070149 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.070267 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:35.071333 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.071871 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.073100 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.073332 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.073399 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.076300 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:35.076302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:35.076303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:35.076314 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:35.077963 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:35.078429 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:35.078430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:35.078431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:35.078496 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:35.078759 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:35.078760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:35.078760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:35.078886 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:35.078969 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.079295 (+ 326us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.079417 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 11:15:35.079771 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.069237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:38619 (local address 127.28.25.1:42931)
0811 11:15:35.069449 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:35.069492 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:35.070009 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:35.070334 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:35.070338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:35.070616 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:35.071231 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.071236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.071987 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.071988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.072992 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.072995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.073473 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.073474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.073602 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.074429 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:35.074438 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:35.076189 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:35.078079 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:35.078090 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:35.078093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:35.078342 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:35.078568 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:35.078570 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:35.078570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:35.078690 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:35.078987 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:35.078996 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:35.079152 (+ 156us) client_negotiation.cc:769] Sending connection context
0811 11:15:35.079271 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 11:15:35.079450 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":124,"threads_started":1}
I20260811 11:15:35.080312 1136 heartbeater.cc:347] Connected to a master server at 127.28.25.62:38619
I20260811 11:15:35.080458 1136 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:35.080662 1136 heartbeater.cc:511] Master 127.28.25.62:38619 requested a full tablet report, sending...
I20260811 11:15:35.081266 903 ts_manager.cc:194] Registered new tserver with Master: bf49e36cb1034d189e58779fea09d36e (127.28.25.1:37897)
I20260811 11:15:35.081926 903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:42931
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:15:35.107733 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:38619
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:35.196334 1142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:35.196594 1142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:35.196626 1142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:35.198009 1142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:35.198072 1142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:35.198104 1142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:35.199846 1142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:38619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:35.200133 1142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:35.200412 1142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:35.206009 1148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:35.206001 1147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:35.206001 1150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:35.206310 1142 server_base.cc:1034] running on GCE node
I20260811 11:15:35.206621 1142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:35.207005 1142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:35.208216 1142 hybrid_clock.cc:648] HybridClock initialized: now 1786446935208165 us; error 71 us; skew 500 ppm
Aug 11 11:15:35 dist-test-slave-2w3w krb5kdc[855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:35.211268 1142 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:15:35.211725 1142 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:35.211809 1142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:35.211979 1142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:35.213227 1142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "a9b55f9027554d8a98caacda2b502679"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "6e0bfeee355db85fd48b73d4146c9b85"
server_key_iv: "3ce2acfbc7d45774a8dc680249123786"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.213634 1142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "a9b55f9027554d8a98caacda2b502679"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "6e0bfeee355db85fd48b73d4146c9b85"
server_key_iv: "3ce2acfbc7d45774a8dc680249123786"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.215162 1142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:15:35.216178 1156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.216410 1142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:35.216531 1142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "a9b55f9027554d8a98caacda2b502679"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "6e0bfeee355db85fd48b73d4146c9b85"
server_key_iv: "3ce2acfbc7d45774a8dc680249123786"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.216615 1142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:35.246237 1142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:35.248965 1142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:35.249166 1142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:35.249470 1142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:35.249836 1142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:35.249897 1142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.249935 1142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:35.249982 1142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.256317 1142 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:36633
I20260811 11:15:35.256356 1282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:36633 every 8 connection(s)
I20260811 11:15:35.256651 1142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
Aug 11 11:15:35 dist-test-slave-2w3w krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:35.264930 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 1142
I20260811 11:15:35.265085 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:35.265343 28772 external_mini_cluster.cc:1428] Setting key 4421d4c41f779275fea159fe3e46b1af
I20260811 11:15:35.268540 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.257441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:57929 (local address 127.28.25.62:38619)
0811 11:15:35.257495 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.257497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.257705 (+ 208us) server_negotiation.cc:407] Connection header received
0811 11:15:35.258260 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.258262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.258274 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.258373 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:35.259663 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.260235 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.261324 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.261538 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.261610 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.264666 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:35.264668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:35.264668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:35.264676 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:35.266847 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:35.267378 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:35.267379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:35.267380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:35.267460 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:35.267733 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:35.267734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:35.267734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:35.267913 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:35.267989 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.268302 (+ 313us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.268432 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:15:35.268792 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.257380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:38619 (local address 127.28.25.2:57929)
0811 11:15:35.257559 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:35.257604 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:35.258138 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:35.258500 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:35.258505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:35.258816 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:35.259531 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.259538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.260361 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.260362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.261215 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.261218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.261691 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.261692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.261825 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.262562 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:35.262572 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:35.264502 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:35.266999 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:35.267012 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:35.267015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:35.267287 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:35.267535 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:35.267536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:35.267536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:35.267646 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:35.267981 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:35.267985 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:35.268121 (+ 136us) client_negotiation.cc:769] Sending connection context
0811 11:15:35.268231 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 11:15:35.268490 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260811 11:15:35.269263 1283 heartbeater.cc:347] Connected to a master server at 127.28.25.62:38619
I20260811 11:15:35.269369 1283 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:35.269582 1283 heartbeater.cc:511] Master 127.28.25.62:38619 requested a full tablet report, sending...
I20260811 11:15:35.270015 903 ts_manager.cc:194] Registered new tserver with Master: a9b55f9027554d8a98caacda2b502679 (127.28.25.2:36633)
I20260811 11:15:35.270409 903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:57929
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:15:35.298705 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:38619
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:35.386346 1289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:35.386579 1289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:35.386606 1289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:35.387912 1289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:35.387964 1289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:35.387992 1289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:35.389672 1289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:38619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:35.389955 1289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:35.390200 1289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:35.395714 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:35.395714 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:35.395740 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:35.396075 1289 server_base.cc:1034] running on GCE node
I20260811 11:15:35.396327 1289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:35.396596 1289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:35.397859 1289 hybrid_clock.cc:648] HybridClock initialized: now 1786446935397821 us; error 56 us; skew 500 ppm
Aug 11 11:15:35 dist-test-slave-2w3w krb5kdc[855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:35.401108 1289 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:15:35.401583 1289 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:35.401646 1289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:35.401793 1289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:35.403115 1289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "403e2123edc9754b9e765f15a4f62b65"
server_key_iv: "42ebd3f28cb72c8262a09aff6ffbbf4a"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.403468 1289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "403e2123edc9754b9e765f15a4f62b65"
server_key_iv: "42ebd3f28cb72c8262a09aff6ffbbf4a"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.405051 1289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:15:35.405925 1304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.406247 1289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:35.406355 1289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "403e2123edc9754b9e765f15a4f62b65"
server_key_iv: "42ebd3f28cb72c8262a09aff6ffbbf4a"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.406482 1289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:35.416121 1289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:35.418608 1289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:35.418776 1289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:35.419106 1289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:35.419462 1289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:35.419517 1289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.419557 1289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:35.419584 1289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.426920 1289 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:36221
I20260811 11:15:35.427070 1438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:36221 every 8 connection(s)
I20260811 11:15:35.427389 1289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
Aug 11 11:15:35 dist-test-slave-2w3w krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:35.435132 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 1289
I20260811 11:15:35.435257 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:15:35.435456 28772 external_mini_cluster.cc:1428] Setting key 6a140b09c7e35f61b45c753f8edc014f
I20260811 11:15:35.438297 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.428210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:37637 (local address 127.28.25.62:38619)
0811 11:15:35.428299 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.428301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.428519 (+ 218us) server_negotiation.cc:407] Connection header received
0811 11:15:35.429047 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.429048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.429059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.429117 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:35.430058 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.430556 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.431924 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.432135 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.432203 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.434735 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:35.434736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:35.434737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:35.434749 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:35.436634 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:35.437201 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:35.437202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:35.437203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:35.437264 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:35.437628 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:35.437629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:35.437629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:35.437771 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:35.437837 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.438152 (+ 315us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.438228 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 11:15:35.438513 1443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.428162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:38619 (local address 127.28.25.3:37637)
0811 11:15:35.428388 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:35.428428 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:35.428947 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:35.429172 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:35.429175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:35.429454 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:35.429963 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.429969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.430698 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.430699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.431806 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.431810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.432226 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.432227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.432348 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.432985 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:35.432993 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:35.434620 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:35.436811 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:35.436821 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:35.436824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:35.437118 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:35.437391 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:35.437392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:35.437392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:35.437528 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:35.437872 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:35.437879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:35.438022 (+ 143us) client_negotiation.cc:769] Sending connection context
0811 11:15:35.438119 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 11:15:35.438271 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":114,"threads_started":1}
I20260811 11:15:35.439375 1439 heartbeater.cc:347] Connected to a master server at 127.28.25.62:38619
I20260811 11:15:35.439532 1439 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:35.439708 1439 heartbeater.cc:511] Master 127.28.25.62:38619 requested a full tablet report, sending...
I20260811 11:15:35.440188 903 ts_manager.cc:194] Registered new tserver with Master: dcff26b2b51e4afab8ba7fe1899f9bc4 (127.28.25.3:36221)
I20260811 11:15:35.440673 903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:37637
I20260811 11:15:35.448146 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:15:35.456286 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.449639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.28.25.62:38619)
0811 11:15:35.449718 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.449720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.449828 (+ 108us) server_negotiation.cc:407] Connection header received
0811 11:15:35.449910 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.449911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.449922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.450001 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:35.450600 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.451208 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.451945 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.452136 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.452248 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.453476 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:35.453477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:35.453477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:35.453485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:35.454943 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:35.455422 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:35.455424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:35.455424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:35.455485 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:35.455691 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:35.455692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:35.455692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:35.455870 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:35.455988 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.456094 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.456181 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:15:35.458169 903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:15:35.460011 903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:35.467625 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.462638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:36221 (local address 127.0.0.1:45870)
0811 11:15:35.463296 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:35.463305 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:35.463355 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:35.463902 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:35.463903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:35.463904 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:35.463938 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.463939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.465056 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.465057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.466509 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.466512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.467285 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.467286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.467372 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.467388 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 11:15:35.467462 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 11:15:35.467527 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":72,"thread_start_us":143,"threads_started":1}
I20260811 11:15:35.467625 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.463152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.28.25.3:36221)
0811 11:15:35.463436 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.463440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.463451 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:15:35.463641 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.463643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.463660 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.463734 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:35.464057 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.464954 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.466616 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.467197 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.467285 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.467358 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.467484 (+ 126us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.467526 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":87,"threads_started":1}
I20260811 11:15:35.467639 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.462629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:36633 (local address 127.0.0.1:36524)
0811 11:15:35.462900 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:35.462987 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:35.463062 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:35.463467 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:35.463471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:35.463474 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:35.463573 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.463575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.465163 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.465164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.466288 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.466291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.467362 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.467364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.467444 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.467450 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:15:35.467486 (+ 36us) client_negotiation.cc:240] Negotiation successful
0811 11:15:35.467539 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":119,"thread_start_us":53,"threads_started":1}
I20260811 11:15:35.468096 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.462774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.28.25.2:36633)
0811 11:15:35.463112 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.463118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.463131 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:15:35.463179 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.463181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.463200 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.463667 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:35.463763 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.465008 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.466395 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.467231 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.467856 (+ 625us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.467936 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.467965 (+ 29us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.468006 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260811 11:15:35.468108 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.463116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.28.25.1:37897)
0811 11:15:35.463343 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.463348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.463501 (+ 153us) server_negotiation.cc:407] Connection header received
0811 11:15:35.463645 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.463647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.463662 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.463734 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:35.464006 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.465218 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.466609 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.467419 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.467492 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.467552 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.467955 (+ 403us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.468003 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260811 11:15:35.467870 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.462951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:37897 (local address 127.0.0.1:35530)
0811 11:15:35.463435 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:35.463447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:35.463505 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:35.463832 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:35.463833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:35.463835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:35.463934 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.463936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.465294 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.465295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.466509 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.466512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.467590 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.467591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.467672 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.467677 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:15:35.467734 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 11:15:35.467791 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":28,"thread_start_us":98,"threads_started":1}
I20260811 11:15:35.468883 1361 tablet_service.cc:1467] Processing CreateTablet for tablet 84c613a5ae494cb5bbd5453a4c61b69a (DEFAULT_TABLE table=test-table [id=3bddf284cbd94938b1e17e46bfee40e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:35.468884 1206 tablet_service.cc:1467] Processing CreateTablet for tablet 84c613a5ae494cb5bbd5453a4c61b69a (DEFAULT_TABLE table=test-table [id=3bddf284cbd94938b1e17e46bfee40e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:35.469148 1048 tablet_service.cc:1467] Processing CreateTablet for tablet 84c613a5ae494cb5bbd5453a4c61b69a (DEFAULT_TABLE table=test-table [id=3bddf284cbd94938b1e17e46bfee40e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:35.469311 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c613a5ae494cb5bbd5453a4c61b69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:35.469326 1361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c613a5ae494cb5bbd5453a4c61b69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:35.469452 1048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c613a5ae494cb5bbd5453a4c61b69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:35.472577 1459 tablet_bootstrap.cc:492] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: Bootstrap starting.
I20260811 11:15:35.472980 1461 tablet_bootstrap.cc:492] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e: Bootstrap starting.
I20260811 11:15:35.473615 1459 tablet_bootstrap.cc:654] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:35.473695 1461 tablet_bootstrap.cc:654] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:35.474434 1461 log.cc:824] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:35.474434 1459 log.cc:824] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:35.474880 1460 tablet_bootstrap.cc:492] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: Bootstrap starting.
I20260811 11:15:35.475544 1461 tablet_bootstrap.cc:492] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e: No bootstrap required, opened a new log
I20260811 11:15:35.475629 1461 ts_tablet_manager.cc:1397] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:15:35.475677 1459 tablet_bootstrap.cc:492] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: No bootstrap required, opened a new log
I20260811 11:15:35.475697 1460 tablet_bootstrap.cc:654] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:35.475765 1459 ts_tablet_manager.cc:1397] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:15:35.476375 1460 log.cc:824] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:35.477360 1460 tablet_bootstrap.cc:492] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: No bootstrap required, opened a new log
I20260811 11:15:35.477448 1460 ts_tablet_manager.cc:1397] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 11:15:35.477798 1459 raft_consensus.cc:348] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 } } peers { permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 } } peers { permanent_uuid: "bf49e36cb1034d189e58779fea09d36e" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37897 } }
I20260811 11:15:35.477983 1459 raft_consensus.cc:374] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:35.478031 1459 raft_consensus.cc:729] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcff26b2b51e4afab8ba7fe1899f9bc4, State: Initialized, Role: FOLLOWER
I20260811 11:15:35.478171 1459 consensus_queue.cc:260] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 } } peers { permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 } } peers { permanent_uuid: "bf49e36cb1034d189e58779fea09d36e" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37897 } }
I20260811 11:15:35.478204 1461 raft_consensus.cc:348] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 } } peers { permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 } } peers { permanent_uuid: "bf49e36cb1034d189e58779fea09d36e" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37897 } }
I20260811 11:15:35.478391 1461 raft_consensus.cc:374] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:35.478440 1461 raft_consensus.cc:729] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf49e36cb1034d189e58779fea09d36e, State: Initialized, Role: FOLLOWER
I20260811 11:15:35.478530 1459 ts_tablet_manager.cc:1428] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:15:35.478587 1461 consensus_queue.cc:260] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 } } peers { permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 } } peers { permanent_uuid: "bf49e36cb1034d189e58779fea09d36e" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37897 } }
I20260811 11:15:35.478679 1439 heartbeater.cc:503] Master 127.28.25.62:38619 was elected leader, sending a full tablet report...
I20260811 11:15:35.478924 1461 ts_tablet_manager.cc:1428] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260811 11:15:35.478933 1136 heartbeater.cc:503] Master 127.28.25.62:38619 was elected leader, sending a full tablet report...
I20260811 11:15:35.480087 1460 raft_consensus.cc:348] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 } } peers { permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 } } peers { permanent_uuid: "bf49e36cb1034d189e58779fea09d36e" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37897 } }
I20260811 11:15:35.480306 1460 raft_consensus.cc:374] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:35.480365 1460 raft_consensus.cc:729] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b55f9027554d8a98caacda2b502679, State: Initialized, Role: FOLLOWER
I20260811 11:15:35.480490 1460 consensus_queue.cc:260] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 } } peers { permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 } } peers { permanent_uuid: "bf49e36cb1034d189e58779fea09d36e" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37897 } }
I20260811 11:15:35.480806 1460 ts_tablet_manager.cc:1428] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260811 11:15:35.480940 1283 heartbeater.cc:503] Master 127.28.25.62:38619 was elected leader, sending a full tablet report...
W20260811 11:15:35.507350 1284 tablet.cc:2369] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:15:35.559299 1468 raft_consensus.cc:482] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:35.559496 1468 raft_consensus.cc:504] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 } } peers { permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 } } peers { permanent_uuid: "bf49e36cb1034d189e58779fea09d36e" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37897 } }
I20260811 11:15:35.560057 1468 leader_election.cc:288] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9b55f9027554d8a98caacda2b502679 (127.28.25.2:36633), dcff26b2b51e4afab8ba7fe1899f9bc4 (127.28.25.3:36221)
I20260811 11:15:35.563589 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.560307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:40971 (local address 127.28.25.2:36633)
0811 11:15:35.560357 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.560360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.560399 (+ 39us) server_negotiation.cc:407] Connection header received
0811 11:15:35.560434 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.560435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.560451 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.560513 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:35.560717 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.561335 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.562579 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.563129 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.563212 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.563240 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.563441 (+ 201us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.563489 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:15:35.563589 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.560223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:36633 (local address 127.28.25.1:40971)
0811 11:15:35.560285 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:35.560299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:35.560353 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:35.560531 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:35.560532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:35.560534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:35.560631 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.560633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.561435 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.561437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.562462 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.562465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.563227 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.563228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.563331 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.563335 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:35.563442 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 11:15:35.563521 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260811 11:15:35.563622 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.560380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:34915 (local address 127.28.25.3:36221)
0811 11:15:35.560436 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.560442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.560509 (+ 67us) server_negotiation.cc:407] Connection header received
0811 11:15:35.560544 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.560545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.560557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.560623 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:35.560874 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.561462 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.562628 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.563163 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.563268 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.563278 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.563472 (+ 194us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.563528 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:15:35.563850 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.560266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:36221 (local address 127.28.25.1:34915)
0811 11:15:35.560418 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:35.560428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:35.560494 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:35.560643 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:35.560643 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:35.560645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:35.560759 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.560761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.561561 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.561562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.562483 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:35.562485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.563296 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.563298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:35.563397 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.563401 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:35.563464 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 11:15:35.563756 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":229,"thread_start_us":73,"threads_started":1}
I20260811 11:15:35.564172 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c613a5ae494cb5bbd5453a4c61b69a" candidate_uuid: "bf49e36cb1034d189e58779fea09d36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b55f9027554d8a98caacda2b502679" is_pre_election: true
I20260811 11:15:35.564224 1382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c613a5ae494cb5bbd5453a4c61b69a" candidate_uuid: "bf49e36cb1034d189e58779fea09d36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" is_pre_election: true
I20260811 11:15:35.564381 1230 raft_consensus.cc:2454] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf49e36cb1034d189e58779fea09d36e in term 0.
I20260811 11:15:35.564402 1382 raft_consensus.cc:2454] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf49e36cb1034d189e58779fea09d36e in term 0.
I20260811 11:15:35.564718 997 leader_election.cc:302] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf49e36cb1034d189e58779fea09d36e, dcff26b2b51e4afab8ba7fe1899f9bc4; no voters:
I20260811 11:15:35.564909 1468 raft_consensus.cc:2781] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:35.564993 1468 raft_consensus.cc:482] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:35.565043 1468 raft_consensus.cc:3037] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:35.565869 1468 raft_consensus.cc:504] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 } } peers { permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 } } peers { permanent_uuid: "bf49e36cb1034d189e58779fea09d36e" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37897 } }
I20260811 11:15:35.566037 1468 leader_election.cc:288] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [CANDIDATE]: Term 1 election: Requested vote from peers a9b55f9027554d8a98caacda2b502679 (127.28.25.2:36633), dcff26b2b51e4afab8ba7fe1899f9bc4 (127.28.25.3:36221)
I20260811 11:15:35.566203 1382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c613a5ae494cb5bbd5453a4c61b69a" candidate_uuid: "bf49e36cb1034d189e58779fea09d36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4"
I20260811 11:15:35.566249 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c613a5ae494cb5bbd5453a4c61b69a" candidate_uuid: "bf49e36cb1034d189e58779fea09d36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b55f9027554d8a98caacda2b502679"
I20260811 11:15:35.566299 1382 raft_consensus.cc:3037] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:35.566342 1230 raft_consensus.cc:3037] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:35.567152 1230 raft_consensus.cc:2454] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf49e36cb1034d189e58779fea09d36e in term 1.
I20260811 11:15:35.567276 1382 raft_consensus.cc:2454] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf49e36cb1034d189e58779fea09d36e in term 1.
I20260811 11:15:35.567390 995 leader_election.cc:302] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9b55f9027554d8a98caacda2b502679, bf49e36cb1034d189e58779fea09d36e; no voters:
I20260811 11:15:35.567507 1468 raft_consensus.cc:2781] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:35.567713 1468 raft_consensus.cc:686] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [term 1 LEADER]: Becoming Leader. State: Replica: bf49e36cb1034d189e58779fea09d36e, State: Running, Role: LEADER
I20260811 11:15:35.567881 1468 consensus_queue.cc:237] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 } } peers { permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 } } peers { permanent_uuid: "bf49e36cb1034d189e58779fea09d36e" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37897 } }
I20260811 11:15:35.568796 903 catalog_manager.cc:5495] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf49e36cb1034d189e58779fea09d36e (127.28.25.1). New cstate: current_term: 1 leader_uuid: "bf49e36cb1034d189e58779fea09d36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf49e36cb1034d189e58779fea09d36e" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37897 } health_report { overall_health: HEALTHY } } }
W20260811 11:15:35.569273 1137 tablet.cc:2369] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:15:35.585027 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.582349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.28.25.1:37897)
0811 11:15:35.582457 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.582460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.582476 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:15:35.582515 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.582516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.582538 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.582612 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:35.582774 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.583374 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.584217 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:35.584421 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:35.584518 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:35.584645 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:35.584753 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:35.584815 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.584913 (+ 98us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.584935 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 11:15:35.587574 1382 raft_consensus.cc:1261] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4 [term 1 FOLLOWER]: Refusing update from remote peer bf49e36cb1034d189e58779fea09d36e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:35.587594 1230 raft_consensus.cc:1261] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679 [term 1 FOLLOWER]: Refusing update from remote peer bf49e36cb1034d189e58779fea09d36e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:35.588157 1471 consensus_queue.cc:1035] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b55f9027554d8a98caacda2b502679" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:35.588322 1468 consensus_queue.cc:1035] T 84c613a5ae494cb5bbd5453a4c61b69a P bf49e36cb1034d189e58779fea09d36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcff26b2b51e4afab8ba7fe1899f9bc4" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:35.592160 1475 mvcc.cc:205] Tried to move back new op lower bound from 7317286648162893824 to 7317286648086773760. Current Snapshot: MvccSnapshot[applied={T|T < 7317286648162893824}]
I20260811 11:15:35.594224 903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:35.594363 903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:35.595340 903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4333a5b1ce8940688ae5ce7079db0e05: Sending DeleteTablet for 3 replicas of tablet 84c613a5ae494cb5bbd5453a4c61b69a
I20260811 11:15:35.595791 1048 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c613a5ae494cb5bbd5453a4c61b69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:35 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:35530
I20260811 11:15:35.595844 1361 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c613a5ae494cb5bbd5453a4c61b69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:35 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:45870
I20260811 11:15:35.596199 1206 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c613a5ae494cb5bbd5453a4c61b69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:35 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:36524
I20260811 11:15:35.596253 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 977
I20260811 11:15:35.598665 1489 tablet_replica.cc:330] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: stopping tablet replica
I20260811 11:15:35.598847 1490 tablet_replica.cc:330] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: stopping tablet replica
I20260811 11:15:35.598857 1489 raft_consensus.cc:2229] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:35.598999 1490 raft_consensus.cc:2229] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:35.599028 1489 raft_consensus.cc:2258] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:35.599116 1490 raft_consensus.cc:2258] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:35.599717 1490 ts_tablet_manager.cc:1905] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:35.600003 1489 ts_tablet_manager.cc:1905] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:35.602210 1490 ts_tablet_manager.cc:1918] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:15:35.602216 1489 ts_tablet_manager.cc:1918] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:15:35.602308 1489 log.cc:1196] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/84c613a5ae494cb5bbd5453a4c61b69a
I20260811 11:15:35.602452 1490 log.cc:1196] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/84c613a5ae494cb5bbd5453a4c61b69a
I20260811 11:15:35.602687 1489 ts_tablet_manager.cc:1939] T 84c613a5ae494cb5bbd5453a4c61b69a P dcff26b2b51e4afab8ba7fe1899f9bc4: Deleting consensus metadata
I20260811 11:15:35.602800 1490 ts_tablet_manager.cc:1939] T 84c613a5ae494cb5bbd5453a4c61b69a P a9b55f9027554d8a98caacda2b502679: Deleting consensus metadata
I20260811 11:15:35.603248 886 catalog_manager.cc:4841] TS dcff26b2b51e4afab8ba7fe1899f9bc4 (127.28.25.3:36221): tablet 84c613a5ae494cb5bbd5453a4c61b69a (table test-table [id=3bddf284cbd94938b1e17e46bfee40e2]) successfully deleted
I20260811 11:15:35.603413 884 catalog_manager.cc:4841] TS a9b55f9027554d8a98caacda2b502679 (127.28.25.2:36633): tablet 84c613a5ae494cb5bbd5453a4c61b69a (table test-table [id=3bddf284cbd94938b1e17e46bfee40e2]) successfully deleted
W20260811 11:15:35.606474 886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:37897 (error 108)
I20260811 11:15:35.606719 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 1142
I20260811 11:15:35.608276 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.608081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:37897 (local address 127.0.0.1:35536)
0811 11:15:35.608161 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:35.608220 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:37897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260811 11:15:35.608450 886 catalog_manager.cc:4584] TS bf49e36cb1034d189e58779fea09d36e (127.28.25.1:37897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84c613a5ae494cb5bbd5453a4c61b69a: Network error: Client connection negotiation failed: client connection to 127.28.25.1:37897: connect: Connection refused (error 111)
I20260811 11:15:35.614323 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 1289
I20260811 11:15:35.622350 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 868
2026-08-11T11:15:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2183 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-11T11:15:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:35Z Disabled control of system clock
I20260811 11:15:35.656852 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:37529
--webserver_interface=127.28.25.62
--webserver_port=0
--builtin_ntp_servers=127.28.25.20:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:37529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 11:15:35.743489 1495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:35.743733 1495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:35.743757 1495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:35.744913 1495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:35.744952 1495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:35.744966 1495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:35.744977 1495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:35.744989 1495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:35.746347 1495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:37529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:37529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:35.746585 1495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:35.746922 1495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:35.752746 1503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:35.752931 1495 server_base.cc:1034] running on GCE node
W20260811 11:15:35.752962 1500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:35.752955 1501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:35.753432 1495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:35.753756 1495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:35.754936 1495 hybrid_clock.cc:648] HybridClock initialized: now 1786446935754914 us; error 40 us; skew 500 ppm
I20260811 11:15:35.762002 1495 webserver.cc:460] Webserver started at http://127.28.25.62:36741/ using document root <none> and password file <none>
I20260811 11:15:35.762334 1495 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:35.762462 1495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:35.762633 1495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:35.763906 1495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "d65f9e7502c544ecb4f7d2b7b19b3223"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "6e3ff20be70c6fa03d511d66930e897f"
server_key_iv: "ced4a4bb2f5fd1896ddae00fe4520683"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.764307 1495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "d65f9e7502c544ecb4f7d2b7b19b3223"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "6e3ff20be70c6fa03d511d66930e897f"
server_key_iv: "ced4a4bb2f5fd1896ddae00fe4520683"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.765784 1495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 11:15:35.766611 1509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.766896 1495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:35.767062 1495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "d65f9e7502c544ecb4f7d2b7b19b3223"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "6e3ff20be70c6fa03d511d66930e897f"
server_key_iv: "ced4a4bb2f5fd1896ddae00fe4520683"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.767153 1495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:35.770713 1495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:35.771071 1495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:35.771236 1495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:35.775485 1495 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:37529
I20260811 11:15:35.775519 1566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:37529 every 8 connection(s)
I20260811 11:15:35.775869 1495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:35.776507 1567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:35.779624 1567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223: Bootstrap starting.
I20260811 11:15:35.780259 1567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:35.780575 1567 log.cc:824] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:35.781378 1567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223: No bootstrap required, opened a new log
I20260811 11:15:35.782356 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 1495
I20260811 11:15:35.782442 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:35.782673 28772 external_mini_cluster.cc:1428] Setting key 4415d821cd26458a177b374cb924a355
I20260811 11:15:35.783382 1567 raft_consensus.cc:348] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d65f9e7502c544ecb4f7d2b7b19b3223" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37529 } }
I20260811 11:15:35.783607 1567 raft_consensus.cc:374] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:35.783681 1567 raft_consensus.cc:729] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d65f9e7502c544ecb4f7d2b7b19b3223, State: Initialized, Role: FOLLOWER
I20260811 11:15:35.783819 1567 consensus_queue.cc:260] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d65f9e7502c544ecb4f7d2b7b19b3223" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37529 } }
I20260811 11:15:35.783910 1567 raft_consensus.cc:388] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:35.783955 1567 raft_consensus.cc:482] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:35.784004 1567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:35.784646 1567 raft_consensus.cc:504] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d65f9e7502c544ecb4f7d2b7b19b3223" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37529 } }
I20260811 11:15:35.784765 1567 leader_election.cc:302] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d65f9e7502c544ecb4f7d2b7b19b3223; no voters:
I20260811 11:15:35.784945 1567 leader_election.cc:288] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:35.785006 1572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:35.785202 1572 raft_consensus.cc:686] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [term 1 LEADER]: Becoming Leader. State: Replica: d65f9e7502c544ecb4f7d2b7b19b3223, State: Running, Role: LEADER
I20260811 11:15:35.785264 1567 sys_catalog.cc:564] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:35.785389 1572 consensus_queue.cc:237] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d65f9e7502c544ecb4f7d2b7b19b3223" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37529 } }
I20260811 11:15:35.785828 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.783390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.28.25.62:37529)
0811 11:15:35.783694 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.783705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.783840 (+ 135us) server_negotiation.cc:407] Connection header received
0811 11:15:35.783996 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.784005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.784058 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.784237 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:35.785031 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:35.785035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:35.785035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:35.785041 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:35.785092 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:35.785108 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:35.785190 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.785289 (+ 99us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.785359 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":109,"threads_started":1}
I20260811 11:15:35.785796 1572 sys_catalog.cc:455] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d65f9e7502c544ecb4f7d2b7b19b3223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d65f9e7502c544ecb4f7d2b7b19b3223" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37529 } } }
I20260811 11:15:35.786010 1572 sys_catalog.cc:458] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:35.785920 1574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d65f9e7502c544ecb4f7d2b7b19b3223. Latest consensus state: current_term: 1 leader_uuid: "d65f9e7502c544ecb4f7d2b7b19b3223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d65f9e7502c544ecb4f7d2b7b19b3223" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37529 } } }
I20260811 11:15:35.786235 1574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:35.786422 1583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:35.787097 1583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:35.788921 1583 catalog_manager.cc:1347] Generated new cluster ID: 961d8b7923154e9f9ee3406bc3612c63
I20260811 11:15:35.788983 1583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:35.822781 1583 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:35.823042 1583 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:35.827654 1583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223: Generated new TSK 0
I20260811 11:15:35.827906 1583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:15:35.837437 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37529
--builtin_ntp_servers=127.28.25.20:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 11:15:35.936173 1596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:35.936411 1596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:35.936439 1596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:35.937657 1596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:35.937698 1596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:35.937727 1596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:35.939168 1596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:35.939393 1596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:35.939680 1596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:35.945189 1604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:35.945283 1603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:35.945464 1596 server_base.cc:1034] running on GCE node
W20260811 11:15:35.945403 1607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:35.945731 1596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:35.946038 1596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:35.947193 1596 hybrid_clock.cc:648] HybridClock initialized: now 1786446935947168 us; error 36 us; skew 500 ppm
I20260811 11:15:35.948768 1596 webserver.cc:460] Webserver started at http://127.28.25.1:46633/ using document root <none> and password file <none>
I20260811 11:15:35.949143 1596 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:35.949201 1596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:35.949361 1596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:35.950583 1596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "fbee066cfbd04374b9a16d9852b1acf4"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "24f8b488daad9731399482514532da2e"
server_key_iv: "59bf8de05a2068d52cef5b9e9c3bf220"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.951036 1596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "fbee066cfbd04374b9a16d9852b1acf4"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "24f8b488daad9731399482514532da2e"
server_key_iv: "59bf8de05a2068d52cef5b9e9c3bf220"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.952550 1596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:35.953382 1614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.953575 1596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:35.953696 1596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "fbee066cfbd04374b9a16d9852b1acf4"
format_stamp: "Formatted at 2026-08-11 11:15:35 on dist-test-slave-2w3w"
server_key: "24f8b488daad9731399482514532da2e"
server_key_iv: "59bf8de05a2068d52cef5b9e9c3bf220"
server_key_version: "encryptionkey@0"
I20260811 11:15:35.953790 1596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:35.957698 1596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:35.958089 1596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:35.958259 1596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:35.958513 1596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:35.958936 1596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:35.959028 1596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.959076 1596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:35.959123 1596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:35.965770 1596 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:42743
I20260811 11:15:35.965852 1739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:42743 every 8 connection(s)
I20260811 11:15:35.966239 1596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:35.970101 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.967456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:38809 (local address 127.28.25.62:37529)
0811 11:15:35.967553 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:35.967555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:35.967774 (+ 219us) server_negotiation.cc:407] Connection header received
0811 11:15:35.968310 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:35.968312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:35.968320 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:35.968417 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:35.969666 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:35.969667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:35.969668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:35.969672 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:35.969690 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:35.969695 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:35.969757 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:35.969946 (+ 189us) server_negotiation.cc:299] Negotiation successful
0811 11:15:35.969965 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 11:15:35.970255 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:35.967380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37529 (local address 127.28.25.1:38809)
0811 11:15:35.967612 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:35.967650 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:35.968191 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:35.968472 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:35.968475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:35.968483 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:35.969450 (+ 967us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:15:35.969450 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:35.969481 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:15:35.969512 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:35.969801 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:35.969808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:35.969811 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:15:35.969906 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 11:15:35.969973 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":114,"threads_started":1}
I20260811 11:15:35.970782 1740 heartbeater.cc:347] Connected to a master server at 127.28.25.62:37529
I20260811 11:15:35.970876 1740 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:35.971086 1740 heartbeater.cc:511] Master 127.28.25.62:37529 requested a full tablet report, sending...
I20260811 11:15:35.971649 1531 ts_manager.cc:194] Registered new tserver with Master: fbee066cfbd04374b9a16d9852b1acf4 (127.28.25.1:42743)
I20260811 11:15:35.974042 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 1596
I20260811 11:15:35.974133 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:35.974355 28772 external_mini_cluster.cc:1428] Setting key 0ed29ea2f087bd1b13bea87b6f18f004
I20260811 11:15:35.975419 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37529
--builtin_ntp_servers=127.28.25.20:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 11:15:36.062440 1743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:36.062680 1743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:36.062708 1743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:36.064023 1743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:36.064080 1743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:36.064109 1743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:36.065452 1743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:36.065687 1743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:36.065979 1743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:15:36.071720 1743 server_base.cc:1034] running on GCE node
W20260811 11:15:36.071683 1752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:36.071683 1749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:36.071910 1748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:36.072376 1743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:36.072650 1743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:36.073864 1743 hybrid_clock.cc:648] HybridClock initialized: now 1786446936073851 us; error 30 us; skew 500 ppm
I20260811 11:15:36.075620 1743 webserver.cc:460] Webserver started at http://127.28.25.2:34105/ using document root <none> and password file <none>
I20260811 11:15:36.075928 1743 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:36.075985 1743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:36.076198 1743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:36.077463 1743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "8560b0ff88524fca98184fa94614c998"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "0ab325fed9213b4371ea1e624f124af7"
server_key_iv: "cf56e5a9cb9e42dfd0e539648d424f92"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.077894 1743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "8560b0ff88524fca98184fa94614c998"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "0ab325fed9213b4371ea1e624f124af7"
server_key_iv: "cf56e5a9cb9e42dfd0e539648d424f92"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.079686 1743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260811 11:15:36.080734 1759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:36.080957 1743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:36.081084 1743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "8560b0ff88524fca98184fa94614c998"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "0ab325fed9213b4371ea1e624f124af7"
server_key_iv: "cf56e5a9cb9e42dfd0e539648d424f92"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.081193 1743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:36.084952 1743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:36.085345 1743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:36.085511 1743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:36.085753 1743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:36.086109 1743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:36.086220 1743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:36.086287 1743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:36.086318 1743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:36.093109 1743 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:39651
I20260811 11:15:36.093155 1883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:39651 every 8 connection(s)
I20260811 11:15:36.093545 1743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:15:36.096984 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.094328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:43621 (local address 127.28.25.62:37529)
0811 11:15:36.094410 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.094412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.094633 (+ 221us) server_negotiation.cc:407] Connection header received
0811 11:15:36.095303 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.095304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.095313 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.095358 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.096670 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.096672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.096672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.096677 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.096698 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.096703 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.096782 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.096886 (+ 104us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.096905 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:15:36.097219 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.094297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37529 (local address 127.28.25.2:43621)
0811 11:15:36.094499 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:36.094538 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:36.095180 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:36.095448 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:36.095452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:36.095468 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:36.096495 (+ 1027us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:15:36.096496 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:36.096527 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:15:36.096559 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:36.096778 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:36.096786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:36.096789 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:15:36.096887 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 11:15:36.096959 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260811 11:15:36.097781 1884 heartbeater.cc:347] Connected to a master server at 127.28.25.62:37529
I20260811 11:15:36.097920 1884 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:36.098062 1884 heartbeater.cc:511] Master 127.28.25.62:37529 requested a full tablet report, sending...
I20260811 11:15:36.098479 1531 ts_manager.cc:194] Registered new tserver with Master: 8560b0ff88524fca98184fa94614c998 (127.28.25.2:39651)
I20260811 11:15:36.101591 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 1743
I20260811 11:15:36.101688 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:36.101939 28772 external_mini_cluster.cc:1428] Setting key 20990fd4f30b11695bc03448653860dd
I20260811 11:15:36.103153 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37529
--builtin_ntp_servers=127.28.25.20:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 11:15:36.194322 1887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:36.194617 1887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:36.194660 1887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:36.196022 1887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:36.196103 1887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:36.196151 1887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:36.197628 1887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:36.197952 1887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:36.198297 1887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:36.204125 1896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:36.204138 1892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:36.204160 1893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:36.204542 1887 server_base.cc:1034] running on GCE node
I20260811 11:15:36.204849 1887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:36.205101 1887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:36.206257 1887 hybrid_clock.cc:648] HybridClock initialized: now 1786446936206232 us; error 38 us; skew 500 ppm
I20260811 11:15:36.207788 1887 webserver.cc:460] Webserver started at http://127.28.25.3:34101/ using document root <none> and password file <none>
I20260811 11:15:36.208060 1887 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:36.208139 1887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:36.208298 1887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:36.209578 1887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "64693562b72e4c2b82f1b4a3f61d99b1"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "82025302f873e5a00580ed815a4f61b7"
server_key_iv: "e82fcc0dc519e34ef33971a4b64c3d7b"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.209973 1887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "64693562b72e4c2b82f1b4a3f61d99b1"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "82025302f873e5a00580ed815a4f61b7"
server_key_iv: "e82fcc0dc519e34ef33971a4b64c3d7b"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.211616 1887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260811 11:15:36.212755 1904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:36.212956 1887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:36.213069 1887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "64693562b72e4c2b82f1b4a3f61d99b1"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "82025302f873e5a00580ed815a4f61b7"
server_key_iv: "e82fcc0dc519e34ef33971a4b64c3d7b"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.213164 1887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:36.216972 1887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:36.217290 1887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:36.217403 1887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:36.217581 1887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:36.217909 1887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:36.217944 1887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:36.217967 1887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:36.217980 1887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:36.224654 1887 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:36063
I20260811 11:15:36.224844 2032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:36063 every 8 connection(s)
I20260811 11:15:36.225153 1887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:15:36.228629 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.225978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:57283 (local address 127.28.25.62:37529)
0811 11:15:36.226023 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.226025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.226278 (+ 253us) server_negotiation.cc:407] Connection header received
0811 11:15:36.226890 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.226892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.226900 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.226980 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.228328 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.228330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.228330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.228336 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.228353 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.228359 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.228434 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.228531 (+ 97us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.228549 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 11:15:36.228897 2035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.225943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37529 (local address 127.28.25.3:57283)
0811 11:15:36.226129 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:36.226174 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:36.226760 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:36.227063 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:36.227067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:36.227075 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:36.228124 (+ 1049us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:15:36.228124 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:36.228157 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:15:36.228191 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:36.228430 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:36.228437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:36.228441 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:36.228544 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 11:15:36.228615 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260811 11:15:36.229357 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 1887
I20260811 11:15:36.229493 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:15:36.229522 2033 heartbeater.cc:347] Connected to a master server at 127.28.25.62:37529
I20260811 11:15:36.229641 2033 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:36.229715 28772 external_mini_cluster.cc:1428] Setting key a8287928d259cf8a2faac7ab70654b9d
I20260811 11:15:36.229887 2033 heartbeater.cc:511] Master 127.28.25.62:37529 requested a full tablet report, sending...
I20260811 11:15:36.230316 1531 ts_manager.cc:194] Registered new tserver with Master: 64693562b72e4c2b82f1b4a3f61d99b1 (127.28.25.3:36063)
I20260811 11:15:36.230764 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:15:36.233441 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.232248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.28.25.62:37529)
0811 11:15:36.232291 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.232293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.232442 (+ 149us) server_negotiation.cc:407] Connection header received
0811 11:15:36.232472 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.232473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.232478 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.232528 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.233200 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.233201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.233201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.233204 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.233220 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.233224 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.233298 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.233310 (+ 12us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.233322 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 11:15:36.235647 1530 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:15:36.237322 1530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:36.241707 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.240068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:36063 (local address 127.0.0.1:46978)
0811 11:15:36.240216 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:36.240263 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:36.240330 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:36.240570 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:36.240577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:36.240585 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:36.241181 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:15:36.241182 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:36.241201 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:15:36.241243 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:36.241531 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:36.241538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:36.241541 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:15:36.241593 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 11:15:36.241609 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260811 11:15:36.242331 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.240358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:39651 (local address 127.0.0.1:34456)
0811 11:15:36.240484 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:36.240493 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:36.240528 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:36.241544 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:36.241545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:36.241547 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:36.241927 (+ 380us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:15:36.241928 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:36.241936 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:15:36.241951 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:36.242175 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:36.242176 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:36.242177 (+ 1us) client_negotiation.cc:769] Sending connection context
0811 11:15:36.242236 (+ 59us) client_negotiation.cc:240] Negotiation successful
0811 11:15:36.242252 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260811 11:15:36.242348 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.241186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.28.25.2:39651)
0811 11:15:36.241369 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.241373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.241385 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:15:36.241414 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.241416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.241425 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.241709 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.242033 (+ 324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.242043 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.242043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.242046 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.242083 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.242095 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.242182 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.242245 (+ 63us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.242261 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260811 11:15:36.242367 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.240180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.28.25.3:36063)
0811 11:15:36.240390 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.240396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.240410 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:15:36.240441 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.240443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.240452 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.240548 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.241352 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.241361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.241361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.241366 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.241407 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.241439 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.242192 (+ 753us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.242234 (+ 42us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.242253 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":137,"threads_started":1}
I20260811 11:15:36.242689 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.240539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:42743 (local address 127.0.0.1:40882)
0811 11:15:36.240644 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:36.240654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:36.240690 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:36.241381 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:36.241382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:36.241385 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:36.242055 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:15:36.242056 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:36.242064 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:15:36.242078 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:36.242581 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:36.242582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:36.242583 (+ 1us) client_negotiation.cc:769] Sending connection context
0811 11:15:36.242615 (+ 32us) client_negotiation.cc:240] Negotiation successful
0811 11:15:36.242631 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":71,"threads_started":1}
I20260811 11:15:36.243093 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.240892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.28.25.1:42743)
0811 11:15:36.241212 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.241217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.241229 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:15:36.241259 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.241261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.241271 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.242159 (+ 888us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.242443 (+ 284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.242446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.242447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.242450 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.242480 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.242497 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.242933 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.242990 (+ 57us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.243009 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":199,"threads_started":1}
I20260811 11:15:36.243130 1810 tablet_service.cc:1467] Processing CreateTablet for tablet dc2a425936bb4bf8bd8b0d158fc75e4a (DEFAULT_TABLE table=test-table [id=78516115189440609d8aa1f46e94c3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:36.243170 1959 tablet_service.cc:1467] Processing CreateTablet for tablet dc2a425936bb4bf8bd8b0d158fc75e4a (DEFAULT_TABLE table=test-table [id=78516115189440609d8aa1f46e94c3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:36.243570 1810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc2a425936bb4bf8bd8b0d158fc75e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:36.243570 1959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc2a425936bb4bf8bd8b0d158fc75e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:36.243836 1669 tablet_service.cc:1467] Processing CreateTablet for tablet dc2a425936bb4bf8bd8b0d158fc75e4a (DEFAULT_TABLE table=test-table [id=78516115189440609d8aa1f46e94c3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:36.244187 1669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc2a425936bb4bf8bd8b0d158fc75e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:36.246240 2051 tablet_bootstrap.cc:492] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: Bootstrap starting.
I20260811 11:15:36.247017 2051 tablet_bootstrap.cc:654] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:36.247164 2053 tablet_bootstrap.cc:492] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: Bootstrap starting.
I20260811 11:15:36.247380 2051 log.cc:824] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:36.247749 2054 tablet_bootstrap.cc:492] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4: Bootstrap starting.
I20260811 11:15:36.248016 2053 tablet_bootstrap.cc:654] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:36.248253 2051 tablet_bootstrap.cc:492] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: No bootstrap required, opened a new log
I20260811 11:15:36.248327 2051 ts_tablet_manager.cc:1397] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:15:36.248392 2053 log.cc:824] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:36.248560 2054 tablet_bootstrap.cc:654] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:36.248922 2054 log.cc:824] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:36.249241 2053 tablet_bootstrap.cc:492] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: No bootstrap required, opened a new log
I20260811 11:15:36.249302 2053 ts_tablet_manager.cc:1397] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:36.249663 2054 tablet_bootstrap.cc:492] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4: No bootstrap required, opened a new log
I20260811 11:15:36.249720 2054 ts_tablet_manager.cc:1397] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:36.251145 2051 raft_consensus.cc:348] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 } } peers { permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 } } peers { permanent_uuid: "fbee066cfbd04374b9a16d9852b1acf4" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42743 } }
I20260811 11:15:36.251358 2051 raft_consensus.cc:374] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:36.251442 2051 raft_consensus.cc:729] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64693562b72e4c2b82f1b4a3f61d99b1, State: Initialized, Role: FOLLOWER
I20260811 11:15:36.251607 2051 consensus_queue.cc:260] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 } } peers { permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 } } peers { permanent_uuid: "fbee066cfbd04374b9a16d9852b1acf4" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42743 } }
I20260811 11:15:36.251947 2051 ts_tablet_manager.cc:1428] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260811 11:15:36.251991 2033 heartbeater.cc:503] Master 127.28.25.62:37529 was elected leader, sending a full tablet report...
I20260811 11:15:36.252199 2053 raft_consensus.cc:348] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 } } peers { permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 } } peers { permanent_uuid: "fbee066cfbd04374b9a16d9852b1acf4" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42743 } }
I20260811 11:15:36.252306 2054 raft_consensus.cc:348] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 } } peers { permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 } } peers { permanent_uuid: "fbee066cfbd04374b9a16d9852b1acf4" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42743 } }
I20260811 11:15:36.252421 2053 raft_consensus.cc:374] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:36.252480 2053 raft_consensus.cc:729] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8560b0ff88524fca98184fa94614c998, State: Initialized, Role: FOLLOWER
I20260811 11:15:36.252492 2054 raft_consensus.cc:374] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:36.252570 2054 raft_consensus.cc:729] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbee066cfbd04374b9a16d9852b1acf4, State: Initialized, Role: FOLLOWER
I20260811 11:15:36.252615 2053 consensus_queue.cc:260] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 } } peers { permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 } } peers { permanent_uuid: "fbee066cfbd04374b9a16d9852b1acf4" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42743 } }
I20260811 11:15:36.252708 2054 consensus_queue.cc:260] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 } } peers { permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 } } peers { permanent_uuid: "fbee066cfbd04374b9a16d9852b1acf4" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42743 } }
I20260811 11:15:36.252916 2053 ts_tablet_manager.cc:1428] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260811 11:15:36.252987 2054 ts_tablet_manager.cc:1428] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260811 11:15:36.253125 1740 heartbeater.cc:503] Master 127.28.25.62:37529 was elected leader, sending a full tablet report...
I20260811 11:15:36.253183 1884 heartbeater.cc:503] Master 127.28.25.62:37529 was elected leader, sending a full tablet report...
I20260811 11:15:36.263337 2060 raft_consensus.cc:482] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:36.263489 2060 raft_consensus.cc:504] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 } } peers { permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 } } peers { permanent_uuid: "fbee066cfbd04374b9a16d9852b1acf4" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42743 } }
I20260811 11:15:36.263818 2060 leader_election.cc:288] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64693562b72e4c2b82f1b4a3f61d99b1 (127.28.25.3:36063), 8560b0ff88524fca98184fa94614c998 (127.28.25.2:39651)
I20260811 11:15:36.265419 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.264165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:37689 (local address 127.28.25.3:36063)
0811 11:15:36.264234 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.264236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.264246 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:15:36.264304 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.264305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.264311 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.264364 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.265151 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.265153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.265153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.265159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.265180 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.265185 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.265251 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.265314 (+ 63us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.265347 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 11:15:36.265419 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.264071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:36063 (local address 127.28.25.1:37689)
0811 11:15:36.264152 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:36.264164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:36.264231 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:36.264405 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:36.264407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:36.264410 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:36.265012 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:15:36.265012 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:36.265023 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:15:36.265041 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:36.265260 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:36.265261 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:36.265262 (+ 1us) client_negotiation.cc:769] Sending connection context
0811 11:15:36.265314 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 11:15:36.265331 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260811 11:15:36.265443 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.264396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:45181 (local address 127.28.25.2:39651)
0811 11:15:36.264447 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.264449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.264458 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:36.264523 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.264523 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.264527 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.264570 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.265165 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.265167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.265167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.265171 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.265188 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.265194 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.265251 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.265366 (+ 115us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.265379 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 11:15:36.265432 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.264208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:39651 (local address 127.28.25.1:45181)
0811 11:15:36.264396 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:36.264407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:36.264458 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:36.264601 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:36.264601 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:36.264603 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:36.265060 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:15:36.265060 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:36.265066 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:15:36.265077 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:36.265260 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:36.265261 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:36.265262 (+ 1us) client_negotiation.cc:769] Sending connection context
0811 11:15:36.265348 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 11:15:36.265359 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":124,"threads_started":1}
I20260811 11:15:36.265799 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc2a425936bb4bf8bd8b0d158fc75e4a" candidate_uuid: "fbee066cfbd04374b9a16d9852b1acf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8560b0ff88524fca98184fa94614c998" is_pre_election: true
I20260811 11:15:36.265914 1981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc2a425936bb4bf8bd8b0d158fc75e4a" candidate_uuid: "fbee066cfbd04374b9a16d9852b1acf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" is_pre_election: true
I20260811 11:15:36.265985 1832 raft_consensus.cc:2454] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbee066cfbd04374b9a16d9852b1acf4 in term 0.
I20260811 11:15:36.266114 1981 raft_consensus.cc:2454] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbee066cfbd04374b9a16d9852b1acf4 in term 0.
I20260811 11:15:36.266252 1615 leader_election.cc:302] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8560b0ff88524fca98184fa94614c998, fbee066cfbd04374b9a16d9852b1acf4; no voters:
I20260811 11:15:36.266414 2060 raft_consensus.cc:2781] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:36.266480 2060 raft_consensus.cc:482] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:36.266521 2060 raft_consensus.cc:3037] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:36.267396 2060 raft_consensus.cc:504] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 } } peers { permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 } } peers { permanent_uuid: "fbee066cfbd04374b9a16d9852b1acf4" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42743 } }
I20260811 11:15:36.267545 2060 leader_election.cc:288] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [CANDIDATE]: Term 1 election: Requested vote from peers 64693562b72e4c2b82f1b4a3f61d99b1 (127.28.25.3:36063), 8560b0ff88524fca98184fa94614c998 (127.28.25.2:39651)
I20260811 11:15:36.267794 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc2a425936bb4bf8bd8b0d158fc75e4a" candidate_uuid: "fbee066cfbd04374b9a16d9852b1acf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8560b0ff88524fca98184fa94614c998"
I20260811 11:15:36.267836 1981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc2a425936bb4bf8bd8b0d158fc75e4a" candidate_uuid: "fbee066cfbd04374b9a16d9852b1acf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64693562b72e4c2b82f1b4a3f61d99b1"
I20260811 11:15:36.267892 1832 raft_consensus.cc:3037] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:36.267922 1981 raft_consensus.cc:3037] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:36.268858 1981 raft_consensus.cc:2454] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbee066cfbd04374b9a16d9852b1acf4 in term 1.
I20260811 11:15:36.268858 1832 raft_consensus.cc:2454] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbee066cfbd04374b9a16d9852b1acf4 in term 1.
I20260811 11:15:36.269068 1615 leader_election.cc:302] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64693562b72e4c2b82f1b4a3f61d99b1, fbee066cfbd04374b9a16d9852b1acf4; no voters:
I20260811 11:15:36.269163 2060 raft_consensus.cc:2781] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:36.269389 2060 raft_consensus.cc:686] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [term 1 LEADER]: Becoming Leader. State: Replica: fbee066cfbd04374b9a16d9852b1acf4, State: Running, Role: LEADER
I20260811 11:15:36.269516 2060 consensus_queue.cc:237] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 } } peers { permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 } } peers { permanent_uuid: "fbee066cfbd04374b9a16d9852b1acf4" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42743 } }
I20260811 11:15:36.270627 1531 catalog_manager.cc:5495] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbee066cfbd04374b9a16d9852b1acf4 (127.28.25.1). New cstate: current_term: 1 leader_uuid: "fbee066cfbd04374b9a16d9852b1acf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbee066cfbd04374b9a16d9852b1acf4" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 42743 } health_report { overall_health: HEALTHY } } }
I20260811 11:15:36.276645 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.275777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.28.25.1:42743)
0811 11:15:36.275871 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.275873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.275881 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:15:36.275915 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.275916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.275929 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.275978 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.276482 (+ 504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.276483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.276483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.276487 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.276501 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.276504 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.276551 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.276581 (+ 30us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.276592 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 11:15:36.278916 1832 raft_consensus.cc:1261] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998 [term 1 FOLLOWER]: Refusing update from remote peer fbee066cfbd04374b9a16d9852b1acf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:36.279073 1981 raft_consensus.cc:1261] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1 [term 1 FOLLOWER]: Refusing update from remote peer fbee066cfbd04374b9a16d9852b1acf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:36.279381 2062 consensus_queue.cc:1035] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8560b0ff88524fca98184fa94614c998" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:36.279541 2060 consensus_queue.cc:1035] T dc2a425936bb4bf8bd8b0d158fc75e4a P fbee066cfbd04374b9a16d9852b1acf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64693562b72e4c2b82f1b4a3f61d99b1" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:36.282770 2067 mvcc.cc:205] Tried to move back new op lower bound from 7317286650994724864 to 7317286650960617472. Current Snapshot: MvccSnapshot[applied={T|T < 7317286650994724864}]
I20260811 11:15:36.285797 1531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:36.285923 1531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:36.286831 1531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d65f9e7502c544ecb4f7d2b7b19b3223: Sending DeleteTablet for 3 replicas of tablet dc2a425936bb4bf8bd8b0d158fc75e4a
I20260811 11:15:36.287287 1959 tablet_service.cc:1514] Processing DeleteTablet for tablet dc2a425936bb4bf8bd8b0d158fc75e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:36 UTC) from {username='slave'} at 127.0.0.1:46978
I20260811 11:15:36.287358 1669 tablet_service.cc:1514] Processing DeleteTablet for tablet dc2a425936bb4bf8bd8b0d158fc75e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:36 UTC) from {username='slave'} at 127.0.0.1:40882
I20260811 11:15:36.287420 1810 tablet_service.cc:1514] Processing DeleteTablet for tablet dc2a425936bb4bf8bd8b0d158fc75e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:36 UTC) from {username='slave'} at 127.0.0.1:34456
I20260811 11:15:36.287622 2076 tablet_replica.cc:330] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: stopping tablet replica
I20260811 11:15:36.287808 2076 raft_consensus.cc:2229] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:36.287825 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 1596
I20260811 11:15:36.287941 2076 raft_consensus.cc:2258] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:36.292908 2078 tablet_replica.cc:330] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: stopping tablet replica
I20260811 11:15:36.293076 2078 raft_consensus.cc:2229] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:36.293224 2078 raft_consensus.cc:2258] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:36.293805 2078 ts_tablet_manager.cc:1905] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:36.293813 2076 ts_tablet_manager.cc:1905] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:36.295475 2076 ts_tablet_manager.cc:1918] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:15:36.295600 2076 log.cc:1196] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/dc2a425936bb4bf8bd8b0d158fc75e4a
I20260811 11:15:36.295871 2076 ts_tablet_manager.cc:1939] T dc2a425936bb4bf8bd8b0d158fc75e4a P 64693562b72e4c2b82f1b4a3f61d99b1: Deleting consensus metadata
I20260811 11:15:36.296005 2078 ts_tablet_manager.cc:1918] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:15:36.296103 2078 log.cc:1196] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/dc2a425936bb4bf8bd8b0d158fc75e4a
I20260811 11:15:36.296409 2078 ts_tablet_manager.cc:1939] T dc2a425936bb4bf8bd8b0d158fc75e4a P 8560b0ff88524fca98184fa94614c998: Deleting consensus metadata
I20260811 11:15:36.296600 1510 catalog_manager.cc:4841] TS 64693562b72e4c2b82f1b4a3f61d99b1 (127.28.25.3:36063): tablet dc2a425936bb4bf8bd8b0d158fc75e4a (table test-table [id=78516115189440609d8aa1f46e94c3c7]) successfully deleted
I20260811 11:15:36.296875 1510 catalog_manager.cc:4841] TS 8560b0ff88524fca98184fa94614c998 (127.28.25.2:39651): tablet dc2a425936bb4bf8bd8b0d158fc75e4a (table test-table [id=78516115189440609d8aa1f46e94c3c7]) successfully deleted
W20260811 11:15:36.298465 1510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:42743 (error 108)
I20260811 11:15:36.299043 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 1743
I20260811 11:15:36.299152 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.298937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:42743 (local address 127.0.0.1:40902)
0811 11:15:36.299041 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:36.299093 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:42743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260811 11:15:36.303172 1510 catalog_manager.cc:4584] TS fbee066cfbd04374b9a16d9852b1acf4 (127.28.25.1:42743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc2a425936bb4bf8bd8b0d158fc75e4a: Network error: Client connection negotiation failed: client connection to 127.28.25.1:42743: connect: Connection refused (error 111)
I20260811 11:15:36.307057 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 1887
I20260811 11:15:36.313812 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 1495
2026-08-11T11:15:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (689 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260811 11:15:36.337321 28772 mini_oidc.cc:152] Starting JWKS server
I20260811 11:15:36.338606 28772 webserver.cc:460] Webserver started at https://127.0.0.1:43607/ using document root <none> and password file <none>
I20260811 11:15:36.338742 28772 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 11:15:36.338948 28772 webserver.cc:460] Webserver started at http://127.0.0.1:41419/ using document root <none> and password file <none>
2026-08-11T11:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:36Z Disabled control of system clock
I20260811 11:15:36.348886 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:37139
--webserver_interface=127.28.25.62
--webserver_port=0
--builtin_ntp_servers=127.28.25.20:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:37139
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43607/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/testchainca.cert with env {}
W20260811 11:15:36.438807 2086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:36.439162 2086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:36.439195 2086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:36.440618 2086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:36.440680 2086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:36.440693 2086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:36.440706 2086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:36.440724 2086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 11:15:36.440737 2086 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 11:15:36.440748 2086 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43607/jwks.json
I20260811 11:15:36.442570 2086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:37139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:37139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43607/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:36.442874 2086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:36.443149 2086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:15:36.448457 2086 server_base.cc:1034] running on GCE node
W20260811 11:15:36.448436 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:36.448426 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:36.448417 2091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:36.449028 2086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:36.449398 2086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:36.450588 2086 hybrid_clock.cc:648] HybridClock initialized: now 1786446936450572 us; error 27 us; skew 500 ppm
I20260811 11:15:36.452152 2086 webserver.cc:460] Webserver started at http://127.28.25.62:38667/ using document root <none> and password file <none>
I20260811 11:15:36.452426 2086 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:36.452505 2086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:36.452663 2086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:36.453819 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "23861ea9bcee444bb05ccda194ac4b7d"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "5bf90333773d261db27fc5785839aae0"
server_key_iv: "734739688605b1e92db4127a868962e2"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.454280 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "23861ea9bcee444bb05ccda194ac4b7d"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "5bf90333773d261db27fc5785839aae0"
server_key_iv: "734739688605b1e92db4127a868962e2"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.455921 2086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260811 11:15:36.456743 2101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:36.456981 2086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:36.457094 2086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "23861ea9bcee444bb05ccda194ac4b7d"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "5bf90333773d261db27fc5785839aae0"
server_key_iv: "734739688605b1e92db4127a868962e2"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.457176 2086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:36.468637 2086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:36.469033 2086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:36.469204 2086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:36.473850 2086 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:37139
I20260811 11:15:36.473873 2156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:37139 every 8 connection(s)
I20260811 11:15:36.474354 2086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:36.475520 2157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:36.478068 2157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d: Bootstrap starting.
I20260811 11:15:36.478742 2157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:36.479131 2157 log.cc:824] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:36.479984 2157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d: No bootstrap required, opened a new log
I20260811 11:15:36.481720 2157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23861ea9bcee444bb05ccda194ac4b7d" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37139 } }
I20260811 11:15:36.481889 2157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:36.481913 2157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23861ea9bcee444bb05ccda194ac4b7d, State: Initialized, Role: FOLLOWER
I20260811 11:15:36.482093 2157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23861ea9bcee444bb05ccda194ac4b7d" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37139 } }
I20260811 11:15:36.482161 2157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:36.482203 2157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:36.482256 2157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:36.482911 2157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23861ea9bcee444bb05ccda194ac4b7d" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37139 } }
I20260811 11:15:36.483062 2157 leader_election.cc:302] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23861ea9bcee444bb05ccda194ac4b7d; no voters:
I20260811 11:15:36.483273 2157 leader_election.cc:288] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:36.483335 2162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:36.483564 2162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [term 1 LEADER]: Becoming Leader. State: Replica: 23861ea9bcee444bb05ccda194ac4b7d, State: Running, Role: LEADER
I20260811 11:15:36.483599 2157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:36.483693 2162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23861ea9bcee444bb05ccda194ac4b7d" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37139 } }
I20260811 11:15:36.484148 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23861ea9bcee444bb05ccda194ac4b7d. Latest consensus state: current_term: 1 leader_uuid: "23861ea9bcee444bb05ccda194ac4b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23861ea9bcee444bb05ccda194ac4b7d" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37139 } } }
I20260811 11:15:36.484252 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:36.484477 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 2086
I20260811 11:15:36.484606 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:36.484833 28772 external_mini_cluster.cc:1428] Setting key 71d329195d170c379855ef52721380ca
I20260811 11:15:36.484236 2163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23861ea9bcee444bb05ccda194ac4b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23861ea9bcee444bb05ccda194ac4b7d" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37139 } } }
I20260811 11:15:36.485069 2163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d [sys.catalog]: This master's current role is: LEADER
W20260811 11:15:36.485671 2180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:15:36.485718 2180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:15:36.485786 2173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:36.485970 2173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:36.487661 2173 catalog_manager.cc:1347] Generated new cluster ID: b87eec4c5a6547a2b49298f26abd369d
I20260811 11:15:36.487737 2173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:36.490716 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.485324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.28.25.62:37139)
0811 11:15:36.485647 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.485656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.485682 (+ 26us) server_negotiation.cc:407] Connection header received
0811 11:15:36.485820 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.485828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.485906 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.486072 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.486319 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.487639 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.488548 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.489058 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.489206 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:36.489975 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.489979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.489979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.490180 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.490222 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.490235 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.490319 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.490405 (+ 86us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.490496 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":79,"threads_started":1}
I20260811 11:15:36.491775 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:15:36.511148 2173 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:36.512189 2173 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:36.520324 2173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23861ea9bcee444bb05ccda194ac4b7d: Generated new TSK 0
I20260811 11:15:36.520565 2173 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 11:15:36.553517 2193 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260811 11:15:36.553651 2193 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260811 11:15:36.554342 2192 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:15:36.530044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37139 (local address 127.0.0.1:45364)
0811 11:15:36.530248 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:36.530262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:36.530353 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:36.530568 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:36.530569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:36.530572 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0811 11:15:36.530639 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:36.530641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.531688 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.531690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:36.532667 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:36.532670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.533166 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.533167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:36.533230 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:36.533257 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 11:15:36.553806 (+ 20549us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 11:15:36.553992 (+ 186us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.25.62:37139: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260811 11:15:36.554821 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.530105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.28.25.62:37139)
0811 11:15:36.530170 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.530172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.530341 (+ 169us) server_negotiation.cc:407] Connection header received
0811 11:15:36.530452 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.530453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.530476 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.530539 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0811 11:15:36.530702 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.531597 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.532808 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.533069 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.533135 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:36.533391 (+ 256us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 11:15:36.553633 (+ 20242us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 11:15:36.554598 (+ 965us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45364: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260811 11:15:36.554893 2183 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45364: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.25.62:37139: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260811 11:15:36.555202 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 2086
2026-08-11T11:15:36Z chronyd exiting
I20260811 11:15:36.596616 28772 test_util.cc:175] -----------------------------------------------
I20260811 11:15:36.596705 28772 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786446911996770-28772-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (260 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260811 11:15:36.597615 28772 mini_oidc.cc:152] Starting JWKS server
I20260811 11:15:36.598490 28772 webserver.cc:460] Webserver started at https://127.0.0.1:37187/ using document root <none> and password file <none>
I20260811 11:15:36.598547 28772 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 11:15:36.598773 28772 webserver.cc:460] Webserver started at http://127.0.0.1:38071/ using document root <none> and password file <none>
2026-08-11T11:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:36Z Disabled control of system clock
I20260811 11:15:36.608786 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:38001
--webserver_interface=127.28.25.62
--webserver_port=0
--builtin_ntp_servers=127.28.25.20:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:38001
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37187/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/testchainca.cert with env {}
W20260811 11:15:36.694798 2200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:36.695128 2200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:36.695176 2200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:36.696534 2200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:36.696624 2200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:36.696647 2200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:36.696671 2200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:36.696700 2200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 11:15:36.696725 2200 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 11:15:36.696750 2200 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37187/jwks.json
I20260811 11:15:36.698664 2200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:38001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:38001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37187/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:36.699021 2200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:36.699276 2200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:36.704907 2209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:36.704880 2206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:36.704887 2207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:36.705152 2200 server_base.cc:1034] running on GCE node
I20260811 11:15:36.705564 2200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:36.705938 2200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:36.707065 2200 hybrid_clock.cc:648] HybridClock initialized: now 1786446936707029 us; error 54 us; skew 500 ppm
I20260811 11:15:36.708712 2200 webserver.cc:460] Webserver started at http://127.28.25.62:36839/ using document root <none> and password file <none>
I20260811 11:15:36.709079 2200 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:36.709136 2200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:36.709301 2200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:36.710546 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "64a4fd72723a40da9b40f6eada03677a"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "71ca3b87f1a1b9dc5fa843894c2d434a"
server_key_iv: "07f661cc68c92256e81fc6a4db2a5b68"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.710948 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "64a4fd72723a40da9b40f6eada03677a"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "71ca3b87f1a1b9dc5fa843894c2d434a"
server_key_iv: "07f661cc68c92256e81fc6a4db2a5b68"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.712719 2200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:36.713946 2215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:36.714274 2200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 11:15:36.714373 2200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "64a4fd72723a40da9b40f6eada03677a"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "71ca3b87f1a1b9dc5fa843894c2d434a"
server_key_iv: "07f661cc68c92256e81fc6a4db2a5b68"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.714493 2200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:36.733628 2200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:36.734055 2200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:36.734220 2200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:36.738976 2200 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:38001
I20260811 11:15:36.739156 2272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:38001 every 8 connection(s)
I20260811 11:15:36.739485 2200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:36.740307 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:36.743238 2273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a: Bootstrap starting.
I20260811 11:15:36.743891 2273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:36.744202 2273 log.cc:824] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:36.744550 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 2200
I20260811 11:15:36.744668 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:36.744861 28772 external_mini_cluster.cc:1428] Setting key 5be011addb8b93f6758269a366076960
I20260811 11:15:36.745162 2273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a: No bootstrap required, opened a new log
I20260811 11:15:36.747170 2273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a4fd72723a40da9b40f6eada03677a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38001 } }
I20260811 11:15:36.747328 2273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:36.747390 2273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a4fd72723a40da9b40f6eada03677a, State: Initialized, Role: FOLLOWER
I20260811 11:15:36.747587 2273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a4fd72723a40da9b40f6eada03677a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38001 } }
I20260811 11:15:36.747696 2273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:36.747747 2273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:36.747802 2273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:36.748441 2273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a4fd72723a40da9b40f6eada03677a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38001 } }
I20260811 11:15:36.748531 2273 leader_election.cc:302] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a4fd72723a40da9b40f6eada03677a; no voters:
I20260811 11:15:36.748742 2273 leader_election.cc:288] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:36.748795 2278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:36.748981 2278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [term 1 LEADER]: Becoming Leader. State: Replica: 64a4fd72723a40da9b40f6eada03677a, State: Running, Role: LEADER
I20260811 11:15:36.749065 2273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:36.749128 2278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a4fd72723a40da9b40f6eada03677a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38001 } }
I20260811 11:15:36.749526 2278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a4fd72723a40da9b40f6eada03677a. Latest consensus state: current_term: 1 leader_uuid: "64a4fd72723a40da9b40f6eada03677a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a4fd72723a40da9b40f6eada03677a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38001 } } }
I20260811 11:15:36.749662 2278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:36.749927 2286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:36.750560 2280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a4fd72723a40da9b40f6eada03677a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a4fd72723a40da9b40f6eada03677a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38001 } } }
I20260811 11:15:36.750669 2280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:36.750790 2286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:36.752430 2286 catalog_manager.cc:1347] Generated new cluster ID: 6501aa5ea26342ed98a1f3d90f901bf3
I20260811 11:15:36.752720 2286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:36.753165 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.745386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.28.25.62:38001)
0811 11:15:36.745630 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.745644 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.745671 (+ 27us) server_negotiation.cc:407] Connection header received
0811 11:15:36.747141 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.747150 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.747227 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.747425 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.747669 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.749372 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.750827 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.751368 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.751517 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:36.752279 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.752282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.752283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.752506 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.752556 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.752568 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.752623 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.752864 (+ 241us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.752950 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
I20260811 11:15:36.754164 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:15:36.764079 2286 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:36.764897 2286 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:36.770228 2286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a4fd72723a40da9b40f6eada03677a: Generated new TSK 0
I20260811 11:15:36.770471 2286 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 11:15:36.793767 2304 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260811 11:15:36.793883 2304 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 11:15:36.794247 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.778815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.28.25.62:38001)
0811 11:15:36.778867 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.778870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.779004 (+ 134us) server_negotiation.cc:407] Connection header received
0811 11:15:36.779139 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.779140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.779154 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.779253 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0811 11:15:36.779410 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.780214 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.781133 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.781467 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.781546 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:36.781723 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 11:15:36.793984 (+ 12261us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 11:15:36.794168 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38314: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":28}
W20260811 11:15:36.794327 2277 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38314: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260811 11:15:36.794374 2303 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:15:36.778728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:38001 (local address 127.0.0.1:38314)
0811 11:15:36.778898 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:36.778910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:36.779049 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:36.779268 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:36.779269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:36.779271 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0811 11:15:36.779334 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:36.779336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.780334 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.780335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:36.781034 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:36.781038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.781557 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.781558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:36.781605 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:36.781609 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 11:15:36.794135 (+ 12526us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 11:15:36.794254 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.25.62:38001: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":112,"threads_started":1}
I20260811 11:15:36.794872 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 2200
2026-08-11T11:15:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260811 11:15:36.835572 28772 mini_oidc.cc:152] Starting JWKS server
I20260811 11:15:36.836299 28772 webserver.cc:460] Webserver started at https://127.0.0.1:34285/ using document root <none> and password file <none>
I20260811 11:15:36.836349 28772 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 11:15:36.836506 28772 webserver.cc:460] Webserver started at http://127.0.0.1:37985/ using document root <none> and password file <none>
2026-08-11T11:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:36Z Disabled control of system clock
I20260811 11:15:36.846377 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39313
--webserver_interface=127.28.25.62
--webserver_port=0
--builtin_ntp_servers=127.28.25.20:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:39313
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34285/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 11:15:36.932677 2311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:36.932991 2311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:36.933038 2311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:36.934316 2311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:36.934393 2311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:36.934414 2311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:36.934438 2311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:36.934471 2311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 11:15:36.934504 2311 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 11:15:36.934526 2311 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34285/jwks.json
I20260811 11:15:36.936087 2311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:39313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34285/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:36.936424 2311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:36.936781 2311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:36.942157 2319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:36.942157 2318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:36.942178 2321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:36.942489 2311 server_base.cc:1034] running on GCE node
I20260811 11:15:36.942744 2311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:36.943125 2311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:36.944353 2311 hybrid_clock.cc:648] HybridClock initialized: now 1786446936944354 us; error 52 us; skew 500 ppm
I20260811 11:15:36.945863 2311 webserver.cc:460] Webserver started at http://127.28.25.62:39667/ using document root <none> and password file <none>
I20260811 11:15:36.946154 2311 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:36.946233 2311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:36.946404 2311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:36.947733 2311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "d6a3235797fd47818a39dccdf6924450"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "8344d7587c21c6b6c019a63b663e9eb1"
server_key_iv: "1831a4bffcd6f355967836f4151895a2"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.948128 2311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "d6a3235797fd47818a39dccdf6924450"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "8344d7587c21c6b6c019a63b663e9eb1"
server_key_iv: "1831a4bffcd6f355967836f4151895a2"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.949599 2311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:15:36.950523 2331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:36.950759 2311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:36.950923 2311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "d6a3235797fd47818a39dccdf6924450"
format_stamp: "Formatted at 2026-08-11 11:15:36 on dist-test-slave-2w3w"
server_key: "8344d7587c21c6b6c019a63b663e9eb1"
server_key_iv: "1831a4bffcd6f355967836f4151895a2"
server_key_version: "encryptionkey@0"
I20260811 11:15:36.951043 2311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:36.964007 2311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:36.964349 2311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:36.964473 2311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:36.968958 2311 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:39313
I20260811 11:15:36.968997 2388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:39313 every 8 connection(s)
I20260811 11:15:36.969390 2311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:36.970101 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:36.972169 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 2311
I20260811 11:15:36.972265 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:36.972493 28772 external_mini_cluster.cc:1428] Setting key a96efd72560bec9cea338c114c14b49b
I20260811 11:15:36.973655 2389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450: Bootstrap starting.
I20260811 11:15:36.974327 2389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:36.974648 2389 log.cc:824] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:36.975484 2389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450: No bootstrap required, opened a new log
I20260811 11:15:36.977363 2389 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a3235797fd47818a39dccdf6924450" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39313 } }
I20260811 11:15:36.977540 2389 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:36.977592 2389 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a3235797fd47818a39dccdf6924450, State: Initialized, Role: FOLLOWER
I20260811 11:15:36.977779 2389 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a3235797fd47818a39dccdf6924450" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39313 } }
I20260811 11:15:36.977886 2389 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:36.977921 2389 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:36.977970 2389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:36.978462 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:36.973151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.28.25.62:39313)
0811 11:15:36.973404 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:36.973413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:36.973455 (+ 42us) server_negotiation.cc:407] Connection header received
0811 11:15:36.973939 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:36.973947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:36.974015 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:36.974183 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:36.974387 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.975497 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.976327 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:36.976776 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:36.976923 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:36.977688 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:36.977691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:36.977692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:36.977926 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:36.977976 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:36.977993 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:36.978081 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:36.978151 (+ 70us) server_negotiation.cc:299] Negotiation successful
0811 11:15:36.978241 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
I20260811 11:15:36.978798 2389 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a3235797fd47818a39dccdf6924450" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39313 } }
I20260811 11:15:36.978926 2389 leader_election.cc:302] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a3235797fd47818a39dccdf6924450; no voters:
I20260811 11:15:36.979169 2389 leader_election.cc:288] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:36.979192 2394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:36.979372 2394 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [term 1 LEADER]: Becoming Leader. State: Replica: d6a3235797fd47818a39dccdf6924450, State: Running, Role: LEADER
I20260811 11:15:36.979460 2389 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:36.979511 2394 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a3235797fd47818a39dccdf6924450" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39313 } }
I20260811 11:15:36.979578 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:15:36.980765 2396 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a3235797fd47818a39dccdf6924450. Latest consensus state: current_term: 1 leader_uuid: "d6a3235797fd47818a39dccdf6924450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a3235797fd47818a39dccdf6924450" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39313 } } }
I20260811 11:15:36.980870 2396 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:36.981072 2395 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6a3235797fd47818a39dccdf6924450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a3235797fd47818a39dccdf6924450" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39313 } } }
I20260811 11:15:36.981179 2395 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450 [sys.catalog]: This master's current role is: LEADER
W20260811 11:15:36.981397 2409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:15:36.981462 2409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:15:36.981530 2412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:36.981709 2412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:36.983309 2412 catalog_manager.cc:1347] Generated new cluster ID: 13d4cb0bbea04af5a459fe488d96c487
I20260811 11:15:36.983385 2412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:36.995679 2412 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:36.996301 2412 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:37.005537 2412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a3235797fd47818a39dccdf6924450: Generated new TSK 0
I20260811 11:15:37.005807 2412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:15:37.023545 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:37.019202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56386 (local address 127.28.25.62:39313)
0811 11:15:37.019278 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:37.019281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:37.019400 (+ 119us) server_negotiation.cc:407] Connection header received
0811 11:15:37.019478 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:37.019479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:37.019492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:37.019596 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0811 11:15:37.019781 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:37.020600 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:37.021553 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:37.021883 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:37.021958 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:37.022170 (+ 212us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 11:15:37.023285 (+ 1115us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 11:15:37.023442 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56386: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260811 11:15:37.023633 2420 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:15:37.019132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39313 (local address 127.0.0.1:56386)
0811 11:15:37.019315 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:37.019331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:37.019406 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:37.019629 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:37.019630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:37.019633 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0811 11:15:37.019699 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:37.019700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:37.020703 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:37.020704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:37.021457 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:37.021459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:37.021981 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:37.021982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:37.022038 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:37.022044 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 11:15:37.023406 (+ 1362us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 11:15:37.023523 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.25.62:39313: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":122,"threads_started":1}
W20260811 11:15:37.024012 2392 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56386: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260811 11:15:37.025498 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 2311
E20260811 11:15:37.036908 2421 webserver.cc:533] Webserver: sslize failed:
E20260811 11:15:37.036994 2421 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-11T11:15:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-11T11:15:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:37Z Disabled control of system clock
I20260811 11:15:37.076205 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33309
--webserver_interface=127.28.25.62
--webserver_port=0
--builtin_ntp_servers=127.28.25.20:39385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:33309
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/testchainca.cert with env {}
W20260811 11:15:37.166217 2425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:37.166460 2425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:37.166486 2425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:37.167726 2425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:37.167773 2425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:37.167785 2425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:37.167799 2425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:37.167811 2425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 11:15:37.167824 2425 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 11:15:37.167835 2425 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260811 11:15:37.169399 2425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:33309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:37.169638 2425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:37.169814 2425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:37.175367 2430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:37.175446 2425 server_base.cc:1034] running on GCE node
W20260811 11:15:37.175395 2433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:37.175401 2431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:37.175942 2425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:37.176282 2425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:37.177450 2425 hybrid_clock.cc:648] HybridClock initialized: now 1786446937177436 us; error 38 us; skew 500 ppm
I20260811 11:15:37.179073 2425 webserver.cc:460] Webserver started at http://127.28.25.62:42515/ using document root <none> and password file <none>
I20260811 11:15:37.179382 2425 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:37.179440 2425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:37.179586 2425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:37.180895 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "0c93a4f509664638815fc9dd9d269894"
format_stamp: "Formatted at 2026-08-11 11:15:37 on dist-test-slave-2w3w"
server_key: "3764f30a3791e519244f3d0e7f5a2132"
server_key_iv: "0ebdb7ff3d56e353df60e9011c45fc48"
server_key_version: "encryptionkey@0"
I20260811 11:15:37.181320 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "0c93a4f509664638815fc9dd9d269894"
format_stamp: "Formatted at 2026-08-11 11:15:37 on dist-test-slave-2w3w"
server_key: "3764f30a3791e519244f3d0e7f5a2132"
server_key_iv: "0ebdb7ff3d56e353df60e9011c45fc48"
server_key_version: "encryptionkey@0"
I20260811 11:15:37.183184 2425 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 11:15:37.184226 2439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:37.184541 2425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:37.184633 2425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "0c93a4f509664638815fc9dd9d269894"
format_stamp: "Formatted at 2026-08-11 11:15:37 on dist-test-slave-2w3w"
server_key: "3764f30a3791e519244f3d0e7f5a2132"
server_key_iv: "0ebdb7ff3d56e353df60e9011c45fc48"
server_key_version: "encryptionkey@0"
I20260811 11:15:37.184741 2425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:37.196774 2425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:37.197178 2425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:37.197337 2425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:37.201829 2425 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:33309
I20260811 11:15:37.201912 2497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:33309 every 8 connection(s)
I20260811 11:15:37.202265 2425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:37.203359 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:37.205803 2498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894: Bootstrap starting.
I20260811 11:15:37.206430 2498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:37.206732 2498 log.cc:824] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:37.207551 2498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894: No bootstrap required, opened a new log
I20260811 11:15:37.209344 2498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c93a4f509664638815fc9dd9d269894" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33309 } }
I20260811 11:15:37.209509 2498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:37.209533 2498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c93a4f509664638815fc9dd9d269894, State: Initialized, Role: FOLLOWER
I20260811 11:15:37.209719 2498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c93a4f509664638815fc9dd9d269894" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33309 } }
I20260811 11:15:37.209784 2498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:37.209854 2498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:37.209918 2498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:37.210551 2498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c93a4f509664638815fc9dd9d269894" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33309 } }
I20260811 11:15:37.210675 2498 leader_election.cc:302] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c93a4f509664638815fc9dd9d269894; no voters:
I20260811 11:15:37.210898 2498 leader_election.cc:288] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:37.211124 2502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:37.211331 2498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:37.211349 2502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [term 1 LEADER]: Becoming Leader. State: Replica: 0c93a4f509664638815fc9dd9d269894, State: Running, Role: LEADER
I20260811 11:15:37.211602 2502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c93a4f509664638815fc9dd9d269894" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33309 } }
I20260811 11:15:37.211887 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 2425
I20260811 11:15:37.212005 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:37.212054 2504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c93a4f509664638815fc9dd9d269894. Latest consensus state: current_term: 1 leader_uuid: "0c93a4f509664638815fc9dd9d269894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c93a4f509664638815fc9dd9d269894" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33309 } } }
I20260811 11:15:37.212147 2504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:37.212237 28772 external_mini_cluster.cc:1428] Setting key 1d4ed9201dbbcf330e65172455700b18
I20260811 11:15:37.212512 2513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:15:37.213416 2503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c93a4f509664638815fc9dd9d269894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c93a4f509664638815fc9dd9d269894" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33309 } } }
I20260811 11:15:37.213539 2503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:37.214439 2513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:37.218238 2513 catalog_manager.cc:1347] Generated new cluster ID: 8af7595a7cba412ab4b3fc95ad4613c1
I20260811 11:15:37.218317 2513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:37.220006 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:37.212944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.28.25.62:33309)
0811 11:15:37.213585 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:37.213597 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:37.213820 (+ 223us) server_negotiation.cc:407] Connection header received
0811 11:15:37.214069 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:37.214077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:37.214185 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:37.214324 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:37.214689 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:37.215815 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:37.217079 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:37.217514 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:37.217613 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:37.219244 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:37.219249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:37.219249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:15:37.219459 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:37.219516 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:15:37.219529 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:37.219610 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:37.219687 (+ 77us) server_negotiation.cc:299] Negotiation successful
0811 11:15:37.219778 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":87,"threads_started":1}
I20260811 11:15:37.221040 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:15:37.227815 2513 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:37.228411 2513 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:37.233668 2513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c93a4f509664638815fc9dd9d269894: Generated new TSK 0
I20260811 11:15:37.233927 2513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:15:37.262140 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:37.245065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.28.25.62:33309)
0811 11:15:37.245126 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:37.245128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:37.245238 (+ 110us) server_negotiation.cc:407] Connection header received
0811 11:15:37.245314 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:37.245315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:37.245330 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:37.245432 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0811 11:15:37.245576 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:37.246569 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:37.247568 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:37.247969 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:37.248058 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:37.248464 (+ 406us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 11:15:37.261784 (+ 13320us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 11:15:37.262042 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39990: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":34}
W20260811 11:15:37.262233 2522 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39990: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260811 11:15:37.262280 2530 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:15:37.244958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33309 (local address 127.0.0.1:39990)
0811 11:15:37.245147 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:37.245160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:37.245237 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:37.245438 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:37.245439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:37.245441 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0811 11:15:37.245503 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:37.245505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:37.246695 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:37.246697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:37.247466 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:37.247470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:37.248247 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:37.248249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:37.248312 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:37.248320 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 11:15:37.261993 (+ 13673us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 11:15:37.262147 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.25.62:33309: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":119,"threads_started":1}
I20260811 11:15:37.262794 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 2425
2026-08-11T11:15:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (215 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (96 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (99 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:37 dist-test-slave-2w3w krb5kdc[2542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:37 dist-test-slave-2w3w krb5kdc[2542](info): set up 2 sockets
Aug 11 11:15:37 dist-test-slave-2w3w krb5kdc[2542](info): commencing operation
krb5kdc: starting...
W20260811 11:15:38.593426 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:38 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:15:38.746896 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:43253
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:38.836510 2559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:38.836812 2559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:38.836859 2559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:38.838203 2559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:38.838297 2559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:38.838320 2559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:38.838343 2559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:38.838369 2559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:38.840119 2559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:43253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:38.840473 2559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:38.840806 2559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:38.846817 2568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:38.846796 2559 server_base.cc:1034] running on GCE node
W20260811 11:15:38.846737 2565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:38.846751 2566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:38.847599 2559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:38.848039 2559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:38.849238 2559 hybrid_clock.cc:648] HybridClock initialized: now 1786446938849215 us; error 49 us; skew 500 ppm
Aug 11 11:15:38 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446938, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:38.852473 2559 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:15:38.853098 2559 webserver.cc:460] Webserver started at http://127.28.25.62:37719/ using document root <none> and password file <none>
I20260811 11:15:38.853410 2559 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:38.853487 2559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:38.853650 2559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:38.854848 2559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "3f0458813af24a3eb56a1ee0b422c745"
format_stamp: "Formatted at 2026-08-11 11:15:38 on dist-test-slave-2w3w"
server_key: "82640281f80e6078dee2e81ba6d44fab"
server_key_iv: "78b9a626e0c65ed5923e31ee6a47d5b5"
server_key_version: "encryptionkey@0"
I20260811 11:15:38.855378 2559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "3f0458813af24a3eb56a1ee0b422c745"
format_stamp: "Formatted at 2026-08-11 11:15:38 on dist-test-slave-2w3w"
server_key: "82640281f80e6078dee2e81ba6d44fab"
server_key_iv: "78b9a626e0c65ed5923e31ee6a47d5b5"
server_key_version: "encryptionkey@0"
I20260811 11:15:38.856977 2559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:15:38.858372 2576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:38.858647 2559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:38.858759 2559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "3f0458813af24a3eb56a1ee0b422c745"
format_stamp: "Formatted at 2026-08-11 11:15:38 on dist-test-slave-2w3w"
server_key: "82640281f80e6078dee2e81ba6d44fab"
server_key_iv: "78b9a626e0c65ed5923e31ee6a47d5b5"
server_key_version: "encryptionkey@0"
I20260811 11:15:38.858871 2559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:38.872159 2559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:38.874457 2559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:38.874642 2559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:38.879271 2559 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:43253
I20260811 11:15:38.879323 2632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:43253 every 8 connection(s)
I20260811 11:15:38.879786 2559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:38.880776 2633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:38.883330 2633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745: Bootstrap starting.
I20260811 11:15:38.883634 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 2559
I20260811 11:15:38.883760 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:38.883963 28772 external_mini_cluster.cc:1428] Setting key a84e28abd2244a52f4c8c2318cfe6581
I20260811 11:15:38.884073 2633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:38.884831 2633 log.cc:824] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:38.885569 2633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745: No bootstrap required, opened a new log
I20260811 11:15:38.887583 2633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0458813af24a3eb56a1ee0b422c745" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43253 } }
I20260811 11:15:38.887749 2633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:38.887825 2633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f0458813af24a3eb56a1ee0b422c745, State: Initialized, Role: FOLLOWER
I20260811 11:15:38.887980 2633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0458813af24a3eb56a1ee0b422c745" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43253 } }
I20260811 11:15:38.888064 2633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:38.888110 2633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:38.888159 2633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:38.888756 2633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0458813af24a3eb56a1ee0b422c745" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43253 } }
I20260811 11:15:38.888883 2633 leader_election.cc:302] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f0458813af24a3eb56a1ee0b422c745; no voters:
I20260811 11:15:38.889106 2633 leader_election.cc:288] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:38.889168 2638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:38.889356 2638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [term 1 LEADER]: Becoming Leader. State: Replica: 3f0458813af24a3eb56a1ee0b422c745, State: Running, Role: LEADER
I20260811 11:15:38.889396 2633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:38.889536 2638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0458813af24a3eb56a1ee0b422c745" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43253 } }
I20260811 11:15:38.889926 2639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f0458813af24a3eb56a1ee0b422c745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0458813af24a3eb56a1ee0b422c745" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43253 } } }
I20260811 11:15:38.890007 2639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:38.890172 2640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f0458813af24a3eb56a1ee0b422c745. Latest consensus state: current_term: 1 leader_uuid: "3f0458813af24a3eb56a1ee0b422c745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0458813af24a3eb56a1ee0b422c745" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 43253 } } }
I20260811 11:15:38.890228 2640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:38.890923 2647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 11 11:15:38 dist-test-slave-2w3w krb5kdc[2542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:38.891697 2647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:38.893445 2647 catalog_manager.cc:1347] Generated new cluster ID: 5cca8ccdcb2d4f7090344c2afffe6515
I20260811 11:15:38.893504 2647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:38.895947 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:38.884560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.28.25.62:43253)
0811 11:15:38.884766 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:38.884776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:38.884812 (+ 36us) server_negotiation.cc:407] Connection header received
0811 11:15:38.885361 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:38.885368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:38.885421 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:38.885576 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:38.886316 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:38.887310 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:38.888235 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:38.888654 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:38.888770 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:38.892172 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:38.892177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:38.892177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:38.892192 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:38.894220 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:38.894627 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:38.894630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:38.894631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:38.894737 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:38.894994 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:38.894995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:38.894995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:38.895277 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:38.895466 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:38.895579 (+ 113us) server_negotiation.cc:299] Negotiation successful
0811 11:15:38.895733 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260811 11:15:38.926646 2647 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:38.927481 2647 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:38.934185 2647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f0458813af24a3eb56a1ee0b422c745: Generated new TSK 0
I20260811 11:15:38.934432 2647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:15:38.975385 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43253
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:39.069864 2663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:39.070101 2663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:39.070129 2663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:39.071518 2663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:39.071593 2663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:39.071662 2663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:39.073366 2663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:39.073668 2663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:39.073947 2663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:39.081741 2668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:39.081849 2672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:39.081750 2670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:39.082540 2663 server_base.cc:1034] running on GCE node
I20260811 11:15:39.082840 2663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:39.083191 2663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:39.084450 2663 hybrid_clock.cc:648] HybridClock initialized: now 1786446939084423 us; error 46 us; skew 500 ppm
Aug 11 11:15:39 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446939, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:39.087749 2663 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:15:39.088343 2663 webserver.cc:460] Webserver started at http://127.28.25.1:41553/ using document root <none> and password file <none>
I20260811 11:15:39.088646 2663 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:39.088727 2663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:39.088912 2663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:39.090305 2663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "9642ccee0976441c9856ad7b22031da5"
format_stamp: "Formatted at 2026-08-11 11:15:39 on dist-test-slave-2w3w"
server_key: "3229a03c600391ac0c903c28a6fe64e0"
server_key_iv: "c736f8b9f858c891664bb266f4fbdbbb"
server_key_version: "encryptionkey@0"
I20260811 11:15:39.090754 2663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "9642ccee0976441c9856ad7b22031da5"
format_stamp: "Formatted at 2026-08-11 11:15:39 on dist-test-slave-2w3w"
server_key: "3229a03c600391ac0c903c28a6fe64e0"
server_key_iv: "c736f8b9f858c891664bb266f4fbdbbb"
server_key_version: "encryptionkey@0"
I20260811 11:15:39.092535 2663 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:39.093643 2681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:39.093918 2663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:39.094046 2663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "9642ccee0976441c9856ad7b22031da5"
format_stamp: "Formatted at 2026-08-11 11:15:39 on dist-test-slave-2w3w"
server_key: "3229a03c600391ac0c903c28a6fe64e0"
server_key_iv: "c736f8b9f858c891664bb266f4fbdbbb"
server_key_version: "encryptionkey@0"
I20260811 11:15:39.094239 2663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:39.101449 2663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:39.104168 2663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:39.104360 2663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:39.104631 2663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:39.104971 2663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:39.105026 2663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:39.105072 2663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:39.105106 2663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:39.111786 2663 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:32857
I20260811 11:15:39.111865 2814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:32857 every 8 connection(s)
I20260811 11:15:39.112222 2663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:39.121937 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 2663
I20260811 11:15:39.122081 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:39.122370 28772 external_mini_cluster.cc:1428] Setting key 18038a164a29bb8626ba16028cd44eca
Aug 11 11:15:39 dist-test-slave-2w3w krb5kdc[2542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446939, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:39.127323 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:39.113211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:48243 (local address 127.28.25.62:43253)
0811 11:15:39.113358 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:39.113365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:39.113564 (+ 199us) server_negotiation.cc:407] Connection header received
0811 11:15:39.114924 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:39.114926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:39.114946 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:39.115128 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:39.116390 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:39.117058 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:39.119226 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:39.119477 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:39.119553 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:39.123659 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:39.123660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:39.123660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:39.123669 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:39.125504 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:39.126134 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:39.126136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:39.126136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:39.126216 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:39.126572 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:39.126572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:39.126573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:39.126697 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:39.126750 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:39.127141 (+ 391us) server_negotiation.cc:299] Negotiation successful
0811 11:15:39.127246 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260811 11:15:39.127635 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:39.113168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:43253 (local address 127.28.25.1:48243)
0811 11:15:39.113360 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:39.113410 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:39.114738 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:39.115231 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:39.115236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:39.115807 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:39.116196 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:39.116204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:39.117198 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:39.117199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:39.119092 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:39.119097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:39.119620 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:39.119621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:39.119773 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:39.120906 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:39.120915 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:39.123521 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:39.125646 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:39.125660 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:39.125664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:39.126054 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:39.126316 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:39.126316 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:39.126317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:39.126504 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:39.126767 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:39.126774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:39.126997 (+ 223us) client_negotiation.cc:769] Sending connection context
0811 11:15:39.127124 (+ 127us) client_negotiation.cc:240] Negotiation successful
0811 11:15:39.127347 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260811 11:15:39.128811 2815 heartbeater.cc:347] Connected to a master server at 127.28.25.62:43253
I20260811 11:15:39.128955 2815 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:39.129202 2815 heartbeater.cc:511] Master 127.28.25.62:43253 requested a full tablet report, sending...
I20260811 11:15:39.129927 2594 ts_manager.cc:194] Registered new tserver with Master: 9642ccee0976441c9856ad7b22031da5 (127.28.25.1:32857)
I20260811 11:15:39.130919 2594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:48243
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
W20260811 11:15:39.894066 28772 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.25.2: real 0.770s user 0.002s sys 0.003s
I20260811 11:15:39.894464 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43253
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:39.985441 2820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:39.985678 2820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:39.985706 2820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:39.987047 2820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:39.987104 2820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:39.987136 2820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:39.988888 2820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:39.989226 2820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:39.989511 2820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:39.995927 2825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:39.995965 2826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:39.995939 2828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:39.996333 2820 server_base.cc:1034] running on GCE node
I20260811 11:15:39.996640 2820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:39.997063 2820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:39.998281 2820 hybrid_clock.cc:648] HybridClock initialized: now 1786446939998259 us; error 35 us; skew 500 ppm
Aug 11 11:15:39 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446939, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:40.001268 2820 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:15:40.001816 2820 webserver.cc:460] Webserver started at http://127.28.25.2:39187/ using document root <none> and password file <none>
I20260811 11:15:40.002027 2820 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:40.002074 2820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:40.002182 2820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:40.003397 2820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "9d62d51aa02b41c5ab431967a364b338"
format_stamp: "Formatted at 2026-08-11 11:15:39 on dist-test-slave-2w3w"
server_key: "d6e3f920f40384f017c7918a29b9babf"
server_key_iv: "a6f6b39cc9515db7f7b92f5325d7d34c"
server_key_version: "encryptionkey@0"
I20260811 11:15:40.003736 2820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "9d62d51aa02b41c5ab431967a364b338"
format_stamp: "Formatted at 2026-08-11 11:15:39 on dist-test-slave-2w3w"
server_key: "d6e3f920f40384f017c7918a29b9babf"
server_key_iv: "a6f6b39cc9515db7f7b92f5325d7d34c"
server_key_version: "encryptionkey@0"
I20260811 11:15:40.005303 2820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:15:40.006485 2837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:40.006831 2820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:40.006978 2820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "9d62d51aa02b41c5ab431967a364b338"
format_stamp: "Formatted at 2026-08-11 11:15:39 on dist-test-slave-2w3w"
server_key: "d6e3f920f40384f017c7918a29b9babf"
server_key_iv: "a6f6b39cc9515db7f7b92f5325d7d34c"
server_key_version: "encryptionkey@0"
I20260811 11:15:40.007113 2820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:40.036465 2820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:40.039130 2820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:40.039319 2820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:40.039629 2820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:40.040042 2820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:40.040104 2820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:40.040140 2820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:40.040171 2820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:40.047250 2820 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:42611
I20260811 11:15:40.047343 2964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:42611 every 8 connection(s)
I20260811 11:15:40.047714 2820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:15:40.051574 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 2820
I20260811 11:15:40.051674 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:40.051900 28772 external_mini_cluster.cc:1428] Setting key fcc9d30ade29aeda3dedbba003939095
Aug 11 11:15:40 dist-test-slave-2w3w krb5kdc[2542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446939, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:40.058229 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:40.048542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:50293 (local address 127.28.25.62:43253)
0811 11:15:40.048685 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:40.048687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:40.048891 (+ 204us) server_negotiation.cc:407] Connection header received
0811 11:15:40.049861 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:40.049863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:40.049875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:40.049955 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:40.050571 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.051177 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.052420 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.052636 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.052714 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:40.055112 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:40.055113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:40.055114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:40.055127 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:40.056684 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:40.057151 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:40.057153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:40.057153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:40.057223 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:40.057501 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:40.057502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:40.057502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:40.057637 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:40.057712 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:40.058022 (+ 310us) server_negotiation.cc:299] Negotiation successful
0811 11:15:40.058123 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260811 11:15:40.058466 2969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:40.048529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:43253 (local address 127.28.25.2:50293)
0811 11:15:40.048725 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:40.048766 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:40.049762 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:40.050012 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:40.050015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:40.050288 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:40.050498 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:40.050502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.051269 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.051270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:40.052272 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:40.052275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.052738 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.052739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:40.052872 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:40.053450 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:40.053459 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:40.054952 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:40.056796 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:40.056805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:40.056808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:40.057064 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:40.057300 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:40.057301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:40.057301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:40.057431 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:40.057713 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:40.057722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:40.057882 (+ 160us) client_negotiation.cc:769] Sending connection context
0811 11:15:40.057997 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 11:15:40.058184 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":126,"threads_started":1}
I20260811 11:15:40.058983 2965 heartbeater.cc:347] Connected to a master server at 127.28.25.62:43253
I20260811 11:15:40.059109 2965 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:40.059314 2965 heartbeater.cc:511] Master 127.28.25.62:43253 requested a full tablet report, sending...
I20260811 11:15:40.059755 2594 ts_manager.cc:194] Registered new tserver with Master: 9d62d51aa02b41c5ab431967a364b338 (127.28.25.2:42611)
I20260811 11:15:40.060343 2594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:50293
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
I20260811 11:15:40.132318 2815 heartbeater.cc:503] Master 127.28.25.62:43253 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
W20260811 11:15:40.795053 28772 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.25.3: real 0.742s user 0.004s sys 0.000s
I20260811 11:15:40.795445 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43253
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:40.884886 2972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:40.885164 2972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:40.885208 2972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:40.886521 2972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:40.886610 2972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:40.886662 2972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:40.888391 2972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:40.888693 2972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:40.888959 2972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:40.894690 2979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:40.894850 2972 server_base.cc:1034] running on GCE node
W20260811 11:15:40.894690 2978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:40.894690 2981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:40.895298 2972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:40.895526 2972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:40.896672 2972 hybrid_clock.cc:648] HybridClock initialized: now 1786446940896638 us; error 47 us; skew 500 ppm
Aug 11 11:15:40 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446940, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:40.905055 2972 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:15:40.905617 2972 webserver.cc:460] Webserver started at http://127.28.25.3:37543/ using document root <none> and password file <none>
I20260811 11:15:40.905885 2972 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:40.905936 2972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:40.906083 2972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:40.907382 2972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "44f120ee9b3246ac8d500e9e0b71d120"
format_stamp: "Formatted at 2026-08-11 11:15:40 on dist-test-slave-2w3w"
server_key: "191b6c562b56f4025d4d4a3365f628d5"
server_key_iv: "33f4ac6da87cd3c6170b7959f5a42190"
server_key_version: "encryptionkey@0"
I20260811 11:15:40.907773 2972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "44f120ee9b3246ac8d500e9e0b71d120"
format_stamp: "Formatted at 2026-08-11 11:15:40 on dist-test-slave-2w3w"
server_key: "191b6c562b56f4025d4d4a3365f628d5"
server_key_iv: "33f4ac6da87cd3c6170b7959f5a42190"
server_key_version: "encryptionkey@0"
I20260811 11:15:40.909277 2972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:15:40.910305 2991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:40.910549 2972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:40.910682 2972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "44f120ee9b3246ac8d500e9e0b71d120"
format_stamp: "Formatted at 2026-08-11 11:15:40 on dist-test-slave-2w3w"
server_key: "191b6c562b56f4025d4d4a3365f628d5"
server_key_iv: "33f4ac6da87cd3c6170b7959f5a42190"
server_key_version: "encryptionkey@0"
I20260811 11:15:40.910841 2972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:40.921998 2972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:40.924605 2972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:40.924749 2972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:40.925089 2972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:40.925485 2972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:40.925546 2972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:40.925582 2972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:40.925612 2972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:40.932381 2972 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:35671
I20260811 11:15:40.932432 3116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:35671 every 8 connection(s)
I20260811 11:15:40.932840 2972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
Aug 11 11:15:40 dist-test-slave-2w3w krb5kdc[2542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446940, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:15:40.942592 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 2972
I20260811 11:15:40.942739 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:15:40.942941 28772 external_mini_cluster.cc:1428] Setting key 3331467c017cde28776760194fdc02ff
I20260811 11:15:40.943506 3119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:40.933695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:45773 (local address 127.28.25.62:43253)
0811 11:15:40.933861 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:40.933864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:40.934073 (+ 209us) server_negotiation.cc:407] Connection header received
0811 11:15:40.935164 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:40.935165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:40.935181 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:40.935250 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:40.935989 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.936528 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.937608 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.937831 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.937927 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:40.940335 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:40.940337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:40.940337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:40.940353 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:40.941809 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:40.942272 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:40.942274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:40.942274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:40.942342 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:40.942641 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:40.942642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:40.942643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:40.942817 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:40.942881 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:40.943290 (+ 409us) server_negotiation.cc:299] Negotiation successful
0811 11:15:40.943404 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260811 11:15:40.943715 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:40.933654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:43253 (local address 127.28.25.3:45773)
0811 11:15:40.933875 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:40.933973 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:40.935035 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:40.935324 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:40.935333 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:40.935617 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:40.935830 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:40.935835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.936647 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.936648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:40.937496 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:40.937499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.937933 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.937934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:40.938082 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:40.938706 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:40.938716 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:40.940216 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:40.941912 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:40.941923 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:40.941926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:40.942183 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:40.942421 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:40.942422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:40.942422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:40.942558 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:40.942889 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:40.942897 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:40.943121 (+ 224us) client_negotiation.cc:769] Sending connection context
0811 11:15:40.943254 (+ 133us) client_negotiation.cc:240] Negotiation successful
0811 11:15:40.943446 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":117,"threads_started":1}
I20260811 11:15:40.944259 3117 heartbeater.cc:347] Connected to a master server at 127.28.25.62:43253
I20260811 11:15:40.944355 3117 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:40.944519 3117 heartbeater.cc:511] Master 127.28.25.62:43253 requested a full tablet report, sending...
I20260811 11:15:40.944859 2594 ts_manager.cc:194] Registered new tserver with Master: 44f120ee9b3246ac8d500e9e0b71d120 (127.28.25.3:35671)
I20260811 11:15:40.945272 2594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:45773
I20260811 11:15:40.955444 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/test-admin.keytab.
Aug 11 11:15:40 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:40.975958 28772 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260811 11:15:40.977887 3125 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:15:40.976931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.28.25.1:32857)
0811 11:15:40.977097 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:40.977102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:40.977113 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:15:40.977141 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:40.977143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:40.977156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:40.977262 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:40.977779 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54760: BlockingRecv error: recv got EOF from 127.0.0.1:54760 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
W20260811 11:15:40.977965 28772 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5cc' of type FILE
Aug 11 11:15:40 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:40.979195 28772 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 11:15:40 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:40.980381 28772 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 11:15:40 dist-test-slave-2w3w krb5kdc[2542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
I20260811 11:15:40.990944 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:40.981152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.28.25.1:32857)
0811 11:15:40.981232 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:40.981234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:40.981332 (+ 98us) server_negotiation.cc:407] Connection header received
0811 11:15:40.981439 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:40.981440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:40.981453 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:40.981515 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:40.981972 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.983094 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.984078 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.984496 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.984596 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:40.987223 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:40.987237 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:40.987237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:40.987248 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:40.989197 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:40.989712 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:40.989716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:40.989717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:40.989875 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:40.990203 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:40.990204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:40.990204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:40.990394 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:40.990544 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:40.990643 (+ 99us) server_negotiation.cc:299] Negotiation successful
0811 11:15:40.990810 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260811 11:15:40.993351 3125 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:15:40.992605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.28.25.1:32857)
0811 11:15:40.992679 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:40.992681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:40.992755 (+ 74us) server_negotiation.cc:407] Connection header received
0811 11:15:40.992823 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:40.992869 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:40.992881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:40.992948 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:40.993295 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54790: BlockingRecv error: recv got EOF from 127.0.0.1:54790 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260811 11:15:40.993450 28772 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5cc' of type FILE
Aug 11 11:15:40 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:40.994541 28772 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 11:15:40 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:40.995657 28772 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 11:15:40 dist-test-slave-2w3w krb5kdc[2542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
I20260811 11:15:41.003800 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:40.996412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.28.25.1:32857)
0811 11:15:40.996495 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:40.996498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:40.996579 (+ 81us) server_negotiation.cc:407] Connection header received
0811 11:15:40.996664 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:40.996665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:40.996675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:40.996748 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:40.997157 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.997720 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.998480 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:40.998698 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:40.998778 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:41.001211 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:41.001212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:41.001212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:41.001221 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:41.002664 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:41.003143 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:41.003144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:41.003144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:41.003209 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:41.003386 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:41.003387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:41.003387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:41.003494 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:41.003584 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:41.003664 (+ 80us) server_negotiation.cc:299] Negotiation successful
0811 11:15:41.003727 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260811 11:15:41.005580 3125 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:15:41.004911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.28.25.1:32857)
0811 11:15:41.004974 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:41.004976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:41.005044 (+ 68us) server_negotiation.cc:407] Connection header received
0811 11:15:41.005111 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:41.005112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:41.005121 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:41.005191 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:41.005508 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54812: BlockingRecv error: recv got EOF from 127.0.0.1:54812 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260811 11:15:41.005673 28772 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786446911996770-28772-0/krb5kdc/krb5cc' of type FILE
Aug 11 11:15:41 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:41.006953 28772 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 11:15:41 dist-test-slave-2w3w krb5kdc[2542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:41.007956 28772 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 11:15:41 dist-test-slave-2w3w krb5kdc[2542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
I20260811 11:15:41.017043 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:41.008528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.28.25.1:32857)
0811 11:15:41.008609 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:41.008611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:41.008666 (+ 55us) server_negotiation.cc:407] Connection header received
0811 11:15:41.008714 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:41.008715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:41.008725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:41.008789 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:41.009420 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:41.009987 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:41.010864 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:41.011161 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:41.011233 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:41.013851 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:41.013852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:41.013853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:41.013861 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:41.015495 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:41.016069 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:41.016070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:41.016070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:41.016141 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:41.016424 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:41.016425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:41.016425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:41.016544 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:41.016622 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:41.016803 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 11:15:41.016877 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 11:15:41.017750 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 2663
I20260811 11:15:41.025485 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 2820
I20260811 11:15:41.032131 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 2972
I20260811 11:15:41.042093 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 2559
2026-08-11T11:15:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3587 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:15:41 dist-test-slave-2w3w krb5kdc[3139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:15:41 dist-test-slave-2w3w krb5kdc[3139](info): set up 2 sockets
Aug 11 11:15:41 dist-test-slave-2w3w krb5kdc[3139](info): commencing operation
krb5kdc: starting...
W20260811 11:15:42.179448 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:15:42 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:15:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:15:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.62.keytab.
Entry for principal oryx/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.62.keytab.
I20260811 11:15:42.319545 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:36971
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.62.keytab
--principal=oryx/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:42.406325 3157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:42.406610 3157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:42.406654 3157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:42.407931 3157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:42.408012 3157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:42.408036 3157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:42.408057 3157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:42.408090 3157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:42.409684 3157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:36971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.62.keytab
--principal=oryx/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:36971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:42.410005 3157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:42.410255 3157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:42.415963 3166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:42.415935 3164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:42.416018 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:42.416312 3157 server_base.cc:1034] running on GCE node
I20260811 11:15:42.416532 3157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:42.416807 3157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:42.418025 3157 hybrid_clock.cc:648] HybridClock initialized: now 1786446942418007 us; error 28 us; skew 500 ppm
Aug 11 11:15:42 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:42.421182 3157 init.cc:377] Logged in from keytab as oryx/127.28.25.62@KRBTEST.COM (short username oryx)
I20260811 11:15:42.421721 3157 webserver.cc:460] Webserver started at http://127.28.25.62:36721/ using document root <none> and password file <none>
I20260811 11:15:42.421977 3157 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:42.422057 3157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:42.422216 3157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:42.423482 3157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "5d9f13a3038847b7a4c0a65ecae990c4"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "6c3ebf5fe4dd45963aa42df9216f038c"
server_key_iv: "69c8a0b8c542f1e457149cd459ab8e31"
server_key_version: "encryptionkey@0"
I20260811 11:15:42.423882 3157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "5d9f13a3038847b7a4c0a65ecae990c4"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "6c3ebf5fe4dd45963aa42df9216f038c"
server_key_iv: "69c8a0b8c542f1e457149cd459ab8e31"
server_key_version: "encryptionkey@0"
I20260811 11:15:42.425460 3157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:15:42.426369 3173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:42.426599 3157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:42.426715 3157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "5d9f13a3038847b7a4c0a65ecae990c4"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "6c3ebf5fe4dd45963aa42df9216f038c"
server_key_iv: "69c8a0b8c542f1e457149cd459ab8e31"
server_key_version: "encryptionkey@0"
I20260811 11:15:42.426800 3157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:42.454557 3157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:42.457551 3157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:42.457733 3157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:42.462348 3157 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:36971
I20260811 11:15:42.462509 3231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:36971 every 8 connection(s)
I20260811 11:15:42.462834 3157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:15:42.464013 3232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:42.466398 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 3157
I20260811 11:15:42.466559 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:15:42.466794 28772 external_mini_cluster.cc:1428] Setting key 46149575cef76fbc108e07d30b4529a6
I20260811 11:15:42.466789 3232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4: Bootstrap starting.
I20260811 11:15:42.467701 3232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:42.468040 3232 log.cc:824] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:42.469002 3232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4: No bootstrap required, opened a new log
I20260811 11:15:42.471415 3232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } }
I20260811 11:15:42.471639 3232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:42.471694 3232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9f13a3038847b7a4c0a65ecae990c4, State: Initialized, Role: FOLLOWER
I20260811 11:15:42.471884 3232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } }
I20260811 11:15:42.471983 3232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:15:42.472010 3232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:15:42.472040 3232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:42.472923 3232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } }
I20260811 11:15:42.473061 3232 leader_election.cc:302] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d9f13a3038847b7a4c0a65ecae990c4; no voters:
I20260811 11:15:42.473273 3232 leader_election.cc:288] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:15:42.473361 3238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:42.473583 3232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:42.473583 3238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5d9f13a3038847b7a4c0a65ecae990c4, State: Running, Role: LEADER
I20260811 11:15:42.473846 3238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } }
I20260811 11:15:42.474253 3240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } }
I20260811 11:15:42.474357 3240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:42.474592 3244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 11 11:15:42 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:15:42.474243 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9f13a3038847b7a4c0a65ecae990c4. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } }
I20260811 11:15:42.475036 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:42.475190 3244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:15:42.478534 3244 catalog_manager.cc:1347] Generated new cluster ID: 5e2eb366ca3d4a04a6b4be00857d13af
I20260811 11:15:42.478597 3244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:15:42.479920 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:42.467501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.28.25.62:36971)
0811 11:15:42.467707 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:42.467723 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:42.467752 (+ 29us) server_negotiation.cc:407] Connection header received
0811 11:15:42.468336 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:42.468344 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:42.468408 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:42.468617 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:42.469485 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:42.470457 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:42.471627 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:42.472408 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:42.472513 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:42.475561 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:42.475566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:42.475566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:42.475581 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:42.477976 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:42.478457 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:42.478461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:42.478461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:42.478599 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:42.478861 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:42.478862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:42.478862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:42.479188 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:42.479404 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:42.479477 (+ 73us) server_negotiation.cc:299] Negotiation successful
0811 11:15:42.479640 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
I20260811 11:15:42.493809 3244 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:15:42.494472 3244 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:15:42.498812 3244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4: Generated new TSK 0
I20260811 11:15:42.499120 3244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab.
Entry for principal oryx/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab.
I20260811 11:15:42.539206 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:36971
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab
--principal=oryx/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:42.650628 3261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:42.650869 3261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:42.650895 3261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:42.652220 3261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:42.652283 3261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:42.652313 3261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:42.653864 3261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab
--principal=oryx/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:42.654103 3261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:42.654309 3261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:42.659876 3266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:42.659931 3267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:42.659910 3269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:42.660058 3261 server_base.cc:1034] running on GCE node
I20260811 11:15:42.660363 3261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:42.660655 3261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:42.661919 3261 hybrid_clock.cc:648] HybridClock initialized: now 1786446942661885 us; error 66 us; skew 500 ppm
Aug 11 11:15:42 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:42.665232 3261 init.cc:377] Logged in from keytab as oryx/127.28.25.1@KRBTEST.COM (short username oryx)
I20260811 11:15:42.665851 3261 webserver.cc:460] Webserver started at http://127.28.25.1:41965/ using document root <none> and password file <none>
I20260811 11:15:42.666179 3261 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:42.666265 3261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:42.666457 3261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:42.667819 3261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "9d3925f34ae44afb9de4bcb773260034"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "0775b7adc704e5ba4b225d73e7c63a68"
server_key_iv: "cc382c6bd05b8312c5839c5abe2f191b"
server_key_version: "encryptionkey@0"
I20260811 11:15:42.668232 3261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "9d3925f34ae44afb9de4bcb773260034"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "0775b7adc704e5ba4b225d73e7c63a68"
server_key_iv: "cc382c6bd05b8312c5839c5abe2f191b"
server_key_version: "encryptionkey@0"
I20260811 11:15:42.670086 3261 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:15:42.671042 3276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:42.671264 3261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:42.671418 3261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "9d3925f34ae44afb9de4bcb773260034"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "0775b7adc704e5ba4b225d73e7c63a68"
server_key_iv: "cc382c6bd05b8312c5839c5abe2f191b"
server_key_version: "encryptionkey@0"
I20260811 11:15:42.671527 3261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:42.689895 3261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:42.692544 3261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:42.692751 3261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:42.693097 3261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:42.693490 3261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:42.693548 3261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:42.693584 3261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:42.693614 3261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:42.700042 3261 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:39279
I20260811 11:15:42.700088 3401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:39279 every 8 connection(s)
I20260811 11:15:42.700448 3261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
Aug 11 11:15:42 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:15:42.707103 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 3261
I20260811 11:15:42.707206 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:15:42.707444 28772 external_mini_cluster.cc:1428] Setting key 2d5f9d87ed2ecf9061087759cdec1042
I20260811 11:15:42.711397 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:42.701241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:54093 (local address 127.28.25.62:36971)
0811 11:15:42.701294 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:42.701297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:42.701553 (+ 256us) server_negotiation.cc:407] Connection header received
0811 11:15:42.702134 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:42.702135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:42.702147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:42.702215 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:42.703366 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:42.703876 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:42.704933 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:42.705138 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:42.705192 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:42.707750 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:42.707751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:42.707752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:42.707760 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:42.709602 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:42.710119 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:42.710120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:42.710120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:42.710191 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:42.710586 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:42.710587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:42.710587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:42.710724 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:42.710818 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:42.711191 (+ 373us) server_negotiation.cc:299] Negotiation successful
0811 11:15:42.711295 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 11:15:42.711699 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:42.701182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:54093)
0811 11:15:42.701397 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:42.701444 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:42.702007 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:42.702317 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:42.702320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:42.702932 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:42.703243 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:42.703250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:42.703972 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:42.703974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:42.704844 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:42.704846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:42.705218 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:42.705219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:42.705347 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:42.705881 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:42.705890 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:42.707635 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:42.709701 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:42.709711 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:42.709713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:42.710035 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:42.710288 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:42.710289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:42.710289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:42.710505 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:42.710799 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:42.710808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:42.711031 (+ 223us) client_negotiation.cc:769] Sending connection context
0811 11:15:42.711166 (+ 135us) client_negotiation.cc:240] Negotiation successful
0811 11:15:42.711365 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
I20260811 11:15:42.712419 3402 heartbeater.cc:347] Connected to a master server at 127.28.25.62:36971
I20260811 11:15:42.712543 3402 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:42.712740 3402 heartbeater.cc:511] Master 127.28.25.62:36971 requested a full tablet report, sending...
I20260811 11:15:42.713366 3192 ts_manager.cc:194] Registered new tserver with Master: 9d3925f34ae44afb9de4bcb773260034 (127.28.25.1:39279)
I20260811 11:15:42.714001 3192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:54093
WARNING: no policy specified for oryx/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab.
Entry for principal oryx/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab.
I20260811 11:15:42.739238 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:36971
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab
--principal=oryx/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:42.827461 3412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:42.827714 3412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:42.827760 3412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:42.829035 3412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:42.829114 3412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:42.829161 3412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:42.830847 3412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab
--principal=oryx/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:42.831163 3412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:42.831429 3412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:42.837215 3418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:42.837337 3417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:42.837337 3420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:42.837728 3412 server_base.cc:1034] running on GCE node
I20260811 11:15:42.838074 3412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:42.838446 3412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:42.839660 3412 hybrid_clock.cc:648] HybridClock initialized: now 1786446942839639 us; error 33 us; skew 500 ppm
Aug 11 11:15:42 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:42.842628 3412 init.cc:377] Logged in from keytab as oryx/127.28.25.2@KRBTEST.COM (short username oryx)
I20260811 11:15:42.843251 3412 webserver.cc:460] Webserver started at http://127.28.25.2:42353/ using document root <none> and password file <none>
I20260811 11:15:42.843510 3412 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:42.843585 3412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:42.843743 3412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:42.844944 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "306af20b690a46bf83a8b269cb11cb81"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "3f7f308da1e127698d434c335d62e6c6"
server_key_iv: "c5cddbfe7b51645670512f5f400ff07f"
server_key_version: "encryptionkey@0"
I20260811 11:15:42.845355 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "306af20b690a46bf83a8b269cb11cb81"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "3f7f308da1e127698d434c335d62e6c6"
server_key_iv: "c5cddbfe7b51645670512f5f400ff07f"
server_key_version: "encryptionkey@0"
I20260811 11:15:42.846939 3412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:15:42.847971 3428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:42.848235 3412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:42.848368 3412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "306af20b690a46bf83a8b269cb11cb81"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "3f7f308da1e127698d434c335d62e6c6"
server_key_iv: "c5cddbfe7b51645670512f5f400ff07f"
server_key_version: "encryptionkey@0"
I20260811 11:15:42.848469 3412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:42.878888 3412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:42.881603 3412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:42.881790 3412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:42.882063 3412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:42.882401 3412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:42.882457 3412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:42.882504 3412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:42.882529 3412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:42.889214 3412 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:36035
I20260811 11:15:42.889264 3561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:36035 every 8 connection(s)
I20260811 11:15:42.889611 3412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:15:42.896229 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 3412
I20260811 11:15:42.896399 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:15:42.896646 28772 external_mini_cluster.cc:1428] Setting key 15551aa78bcb0d43a76966197748ccec
Aug 11 11:15:42 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:15:42.901518 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:42.890405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:52041 (local address 127.28.25.62:36971)
0811 11:15:42.890501 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:42.890503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:42.890790 (+ 287us) server_negotiation.cc:407] Connection header received
0811 11:15:42.891463 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:42.891464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:42.891476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:42.891540 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:42.892524 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:42.893089 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:42.894350 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:42.894550 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:42.894615 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:42.897461 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:42.897463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:42.897463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:42.897472 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:42.899525 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:42.900249 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:42.900251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:42.900252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:42.900333 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:42.900663 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:42.900665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:42.900665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:42.900827 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:42.900925 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:42.901277 (+ 352us) server_negotiation.cc:299] Negotiation successful
0811 11:15:42.901399 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 11:15:42.901795 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:42.890371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:52041)
0811 11:15:42.890568 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:42.890608 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:42.891329 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:42.891596 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:42.891599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:42.892212 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:42.892423 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:42.892428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:42.893211 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:42.893212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:42.894242 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:42.894246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:42.894649 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:42.894650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:42.894840 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:42.895478 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:42.895488 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:42.897339 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:42.899634 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:42.899648 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:42.899652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:42.900123 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:42.900443 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:42.900444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:42.900444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:42.900586 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:42.900914 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:42.900924 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:42.901104 (+ 180us) client_negotiation.cc:769] Sending connection context
0811 11:15:42.901250 (+ 146us) client_negotiation.cc:240] Negotiation successful
0811 11:15:42.901475 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
I20260811 11:15:42.902556 3563 heartbeater.cc:347] Connected to a master server at 127.28.25.62:36971
I20260811 11:15:42.902701 3563 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:42.902912 3563 heartbeater.cc:511] Master 127.28.25.62:36971 requested a full tablet report, sending...
I20260811 11:15:42.903472 3192 ts_manager.cc:194] Registered new tserver with Master: 306af20b690a46bf83a8b269cb11cb81 (127.28.25.2:36035)
I20260811 11:15:42.903895 3192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:52041
WARNING: no policy specified for oryx/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab.
Entry for principal oryx/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab.
I20260811 11:15:42.931802 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:36971
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab
--principal=oryx/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:43.022701 3568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:43.023011 3568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:43.023077 3568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:43.024355 3568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:43.024408 3568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:43.024437 3568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:43.026046 3568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab
--principal=oryx/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:43.026350 3568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:43.026619 3568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:43.032279 3577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.032442 3568 server_base.cc:1034] running on GCE node
W20260811 11:15:43.032382 3573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:43.032279 3575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.032778 3568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:43.033036 3568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:43.034193 3568 hybrid_clock.cc:648] HybridClock initialized: now 1786446943034185 us; error 33 us; skew 500 ppm
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:43.037189 3568 init.cc:377] Logged in from keytab as oryx/127.28.25.3@KRBTEST.COM (short username oryx)
I20260811 11:15:43.037748 3568 webserver.cc:460] Webserver started at http://127.28.25.3:34303/ using document root <none> and password file <none>
I20260811 11:15:43.038010 3568 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:43.038087 3568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:43.038257 3568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:43.039479 3568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "f43c7c3284efd9563f5d84970b667d91"
server_key_iv: "e485236f9cfb2743030d0b30a352705c"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.039871 3568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "f43c7c3284efd9563f5d84970b667d91"
server_key_iv: "e485236f9cfb2743030d0b30a352705c"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.041347 3568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:15:43.042253 3586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.042470 3568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:43.042585 3568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "f43c7c3284efd9563f5d84970b667d91"
server_key_iv: "e485236f9cfb2743030d0b30a352705c"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.042670 3568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:43.066025 3568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:43.068648 3568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:43.068826 3568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:43.069106 3568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:43.069486 3568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:43.069551 3568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.069600 3568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:43.069625 3568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.077710 3568 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:43521
I20260811 11:15:43.077747 3718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:43521 every 8 connection(s)
I20260811 11:15:43.078282 3568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:15:43.078905 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 3568
I20260811 11:15:43.079056 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:15:43.079284 28772 external_mini_cluster.cc:1428] Setting key de165618aec5f37c1577aebd214c57bb
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:15:43.088698 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.079256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:49525 (local address 127.28.25.62:36971)
0811 11:15:43.079302 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.079305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.079585 (+ 280us) server_negotiation.cc:407] Connection header received
0811 11:15:43.080246 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.080248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.080260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.080306 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.081354 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.081869 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.082948 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.083187 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.083279 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.085715 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.085717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.085717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.085725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.087257 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.087691 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.087692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.087692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.087747 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.088026 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.088027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.088027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.088146 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.088202 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.088522 (+ 320us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.088611 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 11:15:43.088831 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.079228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:49525)
0811 11:15:43.079447 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.079504 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.080142 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.080380 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.080384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.081018 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.081241 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.081247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.081974 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.081975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.082825 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.082827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.083295 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.083296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.083440 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.084035 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.084044 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.085598 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.087350 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.087358 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.087360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.087615 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.087832 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.087833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.087833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.087958 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.088207 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.088213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.088345 (+ 132us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.088459 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.088603 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":134,"threads_started":1}
I20260811 11:15:43.089484 3719 heartbeater.cc:347] Connected to a master server at 127.28.25.62:36971
I20260811 11:15:43.089597 3719 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:43.089762 3719 heartbeater.cc:511] Master 127.28.25.62:36971 requested a full tablet report, sending...
I20260811 11:15:43.090214 3192 ts_manager.cc:194] Registered new tserver with Master: 4d17e5e753e64c4faf59dacb5ec6b4ab (127.28.25.3:43521)
I20260811 11:15:43.090611 3192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:49525
I20260811 11:15:43.091498 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.25.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.61.keytab.
Entry for principal oryx/127.28.25.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.61.keytab.
I20260811 11:15:43.123442 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.61:41245
--webserver_interface=127.28.25.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.61.keytab
--principal=oryx/127.28.25.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:36971,127.28.25.61:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:43.215658 3725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:43.215889 3725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:43.215924 3725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:43.217149 3725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:43.217195 3725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:43.217211 3725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:43.217223 3725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:43.217235 3725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:43.218815 3725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:36971,127.28.25.61:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.61.keytab
--principal=oryx/127.28.25.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.61:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.25.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
W20260811 11:15:43.219012 3725 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.25.62:36971,127.28.25.61:41245'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260811 11:15:43.219172 3725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:43.219448 3725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:43.224779 3731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.224998 3725 server_base.cc:1034] running on GCE node
W20260811 11:15:43.224788 3734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:43.224803 3732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.225389 3725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:43.225739 3725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:43.226934 3725 hybrid_clock.cc:648] HybridClock initialized: now 1786446943226920 us; error 35 us; skew 500 ppm
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:43.229890 3725 init.cc:377] Logged in from keytab as oryx/127.28.25.61@KRBTEST.COM (short username oryx)
I20260811 11:15:43.230515 3725 webserver.cc:460] Webserver started at http://127.28.25.61:45841/ using document root <none> and password file <none>
I20260811 11:15:43.230835 3725 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:43.230909 3725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:43.231122 3725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:43.232306 3725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data/instance:
uuid: "6dbb9aa428404b838f3036368e7f4b77"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "0d9914da97592712affa73894a31785b"
server_key_iv: "78e2d28cee7868a8f5a5f90182914498"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.232729 3725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal/instance:
uuid: "6dbb9aa428404b838f3036368e7f4b77"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "0d9914da97592712affa73894a31785b"
server_key_iv: "78e2d28cee7868a8f5a5f90182914498"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.234236 3725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:15:43.235217 3741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.235471 3725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:43.235599 3725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal
uuid: "6dbb9aa428404b838f3036368e7f4b77"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "0d9914da97592712affa73894a31785b"
server_key_iv: "78e2d28cee7868a8f5a5f90182914498"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.235688 3725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:43.252056 3725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:43.254760 3725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:43.255072 3725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:43.261003 3725 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.61:41245
I20260811 11:15:43.261041 3800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.61:41245 every 8 connection(s)
I20260811 11:15:43.261464 3725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
I20260811 11:15:43.262600 3801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:43.264422 3801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:15:43.270387 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 3725
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.61@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:15:43.277390 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.265461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.28.25.62:36971)
0811 11:15:43.265560 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.265563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.265988 (+ 425us) server_negotiation.cc:407] Connection header received
0811 11:15:43.267215 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.267217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.267231 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.267304 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.268304 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.268853 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.269983 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.270197 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.270259 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.273393 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.273395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.273395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.273403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.275454 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.276126 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.276128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.276129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.276192 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.276466 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.276468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.276468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.276715 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.276813 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.277153 (+ 340us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.277254 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 11:15:43.277477 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.265300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:56022)
0811 11:15:43.265757 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.265801 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.267041 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.267403 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.267408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.267799 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.268183 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.268188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.268983 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.268985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.269874 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.269876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.270294 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.270296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.270472 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.271249 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.271260 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.273274 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.275588 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.275596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.275598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.276050 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.276270 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.276271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.276271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.276389 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.276821 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.276828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.276999 (+ 171us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.277101 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.277251 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":166,"threads_started":1}
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.25.61@KRBTEST.COM
I20260811 11:15:43.278309 3801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:15:43.281721 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.271122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.28.25.61:41245)
0811 11:15:43.271365 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.271371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.271444 (+ 73us) server_negotiation.cc:407] Connection header received
0811 11:15:43.271492 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.271495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.271519 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.271598 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.272449 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.273448 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.274622 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.275188 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.275253 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.278467 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.278475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.278476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.278485 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.280304 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.280744 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.280747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.280748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.280815 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.281030 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.281031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.281031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.281186 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.281364 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.281440 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.281627 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":138,"threads_started":1}
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.61@KRBTEST.COM for oryx/127.28.25.61@KRBTEST.COM
I20260811 11:15:43.286226 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.278624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41245 (local address 127.0.0.1:40182)
0811 11:15:43.278678 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.278689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.278724 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.278996 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.278997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.279148 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.279196 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.279198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.279899 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.279900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.280386 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.280387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.280809 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.280810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.280883 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.281427 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.281431 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.282903 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.285151 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.285152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.285152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.285328 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.285535 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.285536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.285536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.285591 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.285865 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.285866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.285906 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.285975 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.286112 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 11:15:43.286307 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.278686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.28.25.61:41245)
0811 11:15:43.278866 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.278867 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.278873 (+ 6us) server_negotiation.cc:407] Connection header received
0811 11:15:43.278887 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.278887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.278895 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.278976 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.279292 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.279811 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.280485 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.280713 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.280809 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.283018 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.283019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.283019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.283026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.285043 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.285403 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.285404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.285404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.285467 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.285669 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.285670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.285670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.285795 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.285875 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.286003 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.286167 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":156,"thread_start_us":122,"threads_started":1}
I20260811 11:15:43.288131 3801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: Bootstrap starting.
I20260811 11:15:43.288838 3801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:43.289144 3801 log.cc:824] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:43.290011 3801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: No bootstrap required, opened a new log
I20260811 11:15:43.291586 3801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } }
I20260811 11:15:43.291745 3801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:43.291796 3801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbb9aa428404b838f3036368e7f4b77, State: Initialized, Role: FOLLOWER
I20260811 11:15:43.291942 3801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } }
I20260811 11:15:43.292289 3801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:43.292296 3809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } } }
I20260811 11:15:43.292470 3809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 11:15:43.294417 3823 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:15:43.294503 3823 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:15:43.296526 3725 master_runner.cc:428] Detected that this master 6dbb9aa428404b838f3036368e7f4b77 is joining an existing cluster
I20260811 11:15:43.296623 3725 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.25.61:41245
I20260811 11:15:43.304905 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.297883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.28.25.62:36971)
0811 11:15:43.297931 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.297933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.298034 (+ 101us) server_negotiation.cc:407] Connection header received
0811 11:15:43.298116 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.298117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.298136 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.298203 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.298606 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.299234 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.300036 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.300227 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.300301 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.301765 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.301767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.301767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.301776 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.303564 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.304048 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.304050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.304050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.304120 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.304311 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.304312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.304313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.304448 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.304545 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.304710 (+ 165us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.304791 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:15:43.305004 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.297833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.28.25.61:41245)
0811 11:15:43.297880 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.297882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.298128 (+ 246us) server_negotiation.cc:407] Connection header received
0811 11:15:43.298217 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.298218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.298226 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.298253 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.298725 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.299494 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.300519 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.300744 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.300838 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.301860 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.301861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.301861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.301868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.303569 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.304124 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.304125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.304125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.304176 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.304356 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.304357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.304357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.304494 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.304564 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.304720 (+ 156us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.304878 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":21}
I20260811 11:15:43.305176 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.297770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:56032)
0811 11:15:43.297975 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.297986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.298050 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.298208 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.298209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.298444 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.298523 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.298525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.299355 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.299356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.299933 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.299935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.300343 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.300345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.300404 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.301065 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.301069 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.301662 (+ 593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.303687 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.303688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.303689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.303961 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.304188 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.304188 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.304189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.304258 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.304537 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.304538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.304585 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.304667 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.304803 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":101,"threads_started":1}
I20260811 11:15:43.305310 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.297770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41245 (local address 127.0.0.1:40198)
0811 11:15:43.298060 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.298071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.298151 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.298307 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.298308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.298566 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.298642 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.298644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.299582 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.299583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.300434 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.300437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.300844 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.300845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.300913 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.301299 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.301304 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.301753 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.303703 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.303704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.303704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.304050 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.304234 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.304235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.304235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.304295 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.304587 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.304588 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.304624 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.304752 (+ 128us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.305179 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":356,"thread_start_us":199,"threads_started":1}
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.62@KRBTEST.COM for oryx/127.28.25.61@KRBTEST.COM
I20260811 11:15:43.316735 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.307987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41245 (local address 127.0.0.1:40200)
0811 11:15:43.308171 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.308288 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.308379 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.308539 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.308542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.308823 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.308945 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.308948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.309948 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.309949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.310585 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.310588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.311100 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.311101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.311180 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.311758 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.311769 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.313370 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.315456 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.315465 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.315468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.315707 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.315937 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.315938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.315938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.316009 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.316336 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.316344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.316407 (+ 63us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.316509 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.316637 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":130,"threads_started":1}
I20260811 11:15:43.316747 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.308062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.28.25.61:41245)
0811 11:15:43.308107 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.308110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.308374 (+ 264us) server_negotiation.cc:407] Connection header received
0811 11:15:43.308454 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.308455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.308466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.308535 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.309026 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.309815 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.310673 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.311017 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.311107 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.313489 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.313490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.313491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.313500 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.315318 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.315796 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.315797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.315797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.315865 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.316074 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.316075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.316076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.316239 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.316318 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.316551 (+ 233us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.316648 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 11:15:43.317585 3192 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.25.61:41245: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77"
member_type: NON_VOTER
last_known_addr {
host: "127.28.25.61"
port: 41245
}
attrs {
promote: true
}
}
dest_uuid: "5d9f13a3038847b7a4c0a65ecae990c4"
cas_config_opid_index: -1
I20260811 11:15:43.318022 3192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } }
I20260811 11:15:43.318568 3834 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6dbb9aa428404b838f3036368e7f4b77 (127.28.25.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } }
I20260811 11:15:43.319089 3772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:43.319900 3834 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.25.61:41245
I20260811 11:15:43.320015 3834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } }
I20260811 11:15:43.320092 3834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:43.320156 3772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Refusing update from remote peer 5d9f13a3038847b7a4c0a65ecae990c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260811 11:15:43.320250 3725 master.cc:537] Master@127.28.25.61:41245 shutting down...
I20260811 11:15:43.320992 3833 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 11:15:43.322320 3175 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260811 11:15:43.322883 3175 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 -> Peer 6dbb9aa428404b838f3036368e7f4b77 (127.28.25.61:41245): Couldn't send request to peer 6dbb9aa428404b838f3036368e7f4b77. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 11:15:43.323684 3725 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:43.323788 3725 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:43.323854 3725 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: stopping tablet replica
I20260811 11:15:43.337796 3725 master.cc:552] Master@127.28.25.61:41245 shutdown complete.
I20260811 11:15:43.337919 3725 master_runner.cc:305] Clearing existing system tablet
I20260811 11:15:43.338094 3725 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:43.338145 3725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:43.338850 3725 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.001s
I20260811 11:15:43.339515 3839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.339648 3725 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:43.339707 3725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.339730 3725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal
uuid: "6dbb9aa428404b838f3036368e7f4b77"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "0d9914da97592712affa73894a31785b"
server_key_iv: "78e2d28cee7868a8f5a5f90182914498"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.340509 3725 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:43.342216 3725 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 11:15:43.342294 3725 log.cc:1196] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260811 11:15:43.342538 3725 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: Deleting consensus metadata
I20260811 11:15:43.342712 3725 master_runner.cc:315] Copying system tablet from 127.28.25.62:36971
I20260811 11:15:43.343523 3725 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: tablet copy: Beginning tablet copy session from remote peer at address 127.28.25.62:36971
I20260811 11:15:43.351253 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.343953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:56046)
0811 11:15:43.344143 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.344159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.344227 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.344443 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.344444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.344614 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.344680 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.344682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.345700 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.345701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.346316 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.346318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.346953 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.346954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.347114 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.347772 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.347802 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.348148 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.350070 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.350071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.350071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.350266 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.350489 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.350490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.350490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.350548 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.350874 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.350875 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.350923 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.351063 (+ 140us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.351167 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260811 11:15:43.351279 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.344069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.28.25.62:36971)
0811 11:15:43.344139 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.344141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.344221 (+ 80us) server_negotiation.cc:407] Connection header received
0811 11:15:43.344305 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.344307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.344319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.344388 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.344769 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.345579 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.346469 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.346780 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.346858 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.348239 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.348240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.348241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.348249 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.349972 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.350341 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.350342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.350342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.350418 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.350620 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.350621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.350621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.350767 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.350838 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.351094 (+ 256us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.351184 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 11:15:43.351794 3214 tablet_copy_service.cc:140] P 5d9f13a3038847b7a4c0a65ecae990c4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6dbb9aa428404b838f3036368e7f4b77 ({username='oryx', principal='oryx/127.28.25.61@KRBTEST.COM'} at 127.0.0.1:56046)
I20260811 11:15:43.351972 3214 tablet_copy_service.cc:161] P 5d9f13a3038847b7a4c0a65ecae990c4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6dbb9aa428404b838f3036368e7f4b77 at {username='oryx', principal='oryx/127.28.25.61@KRBTEST.COM'} at 127.0.0.1:56046: session id = 6dbb9aa428404b838f3036368e7f4b77-00000000000000000000000000000000
I20260811 11:15:43.352754 3214 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 11:15:43.353515 3725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 11:15:43.355125 3725 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: tablet copy: Starting download of 0 data blocks...
I20260811 11:15:43.355300 3725 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: tablet copy: Starting download of 1 WAL segments...
I20260811 11:15:43.356896 3725 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 11:15:43.358233 3214 tablet_copy_service.cc:342] P 5d9f13a3038847b7a4c0a65ecae990c4: Request end of tablet copy session 6dbb9aa428404b838f3036368e7f4b77-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.25.61@KRBTEST.COM'} at 127.0.0.1:56046
I20260811 11:15:43.358328 3214 tablet_copy_service.cc:433] P 5d9f13a3038847b7a4c0a65ecae990c4: ending tablet copy session 6dbb9aa428404b838f3036368e7f4b77-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6dbb9aa428404b838f3036368e7f4b77
I20260811 11:15:43.361545 3725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:43.363153 3848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:43.363384 3851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:43.363586 3849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.366876 3725 server_base.cc:1034] running on GCE node
I20260811 11:15:43.367050 3725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:43.367242 3725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:43.368391 3725 hybrid_clock.cc:648] HybridClock initialized: now 1786446943368385 us; error 31 us; skew 500 ppm
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:43.369911 3725 init.cc:377] Logged in from keytab as oryx/127.28.25.61@KRBTEST.COM (short username oryx)
I20260811 11:15:43.370260 3725 webserver.cc:460] Webserver started at http://127.28.25.61:36133/ using document root <none> and password file <none>
I20260811 11:15:43.370404 3725 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:43.370455 3725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:43.371169 3725 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260811 11:15:43.371755 3859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.371977 3725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260811 11:15:43.372056 3725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal
uuid: "6dbb9aa428404b838f3036368e7f4b77"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "0d9914da97592712affa73894a31785b"
server_key_iv: "78e2d28cee7868a8f5a5f90182914498"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.372121 3725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:43.400614 3725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:43.402493 3725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:43.405730 3725 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.61:41245
I20260811 11:15:43.405771 3915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.61:41245 every 8 connection(s)
I20260811 11:15:43.406106 3725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
I20260811 11:15:43.406929 3916 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:15:43.407415 3916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: Bootstrap starting.
I20260811 11:15:43.410123 3916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:15:43.410384 3916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: Bootstrap complete.
I20260811 11:15:43.410597 3916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } }
I20260811 11:15:43.410673 3916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dbb9aa428404b838f3036368e7f4b77, State: Initialized, Role: LEARNER
I20260811 11:15:43.410702 3916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } }
I20260811 11:15:43.410892 3916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:43.410916 3919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } }
I20260811 11:15:43.411056 3919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: This master's current role is: LEARNER
I20260811 11:15:43.413307 3931 catalog_manager.cc:1259] Loaded cluster ID: 5e2eb366ca3d4a04a6b4be00857d13af
I20260811 11:15:43.413373 3931 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: loading cluster ID for follower catalog manager: success
I20260811 11:15:43.414114 3931 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: acquiring CA information for follower catalog manager: success
I20260811 11:15:43.414407 3931 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:15:43.442092 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.434218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.28.25.61:41245)
0811 11:15:43.434408 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.434410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.434421 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:15:43.434456 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.434456 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.434469 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.434564 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.435224 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.436232 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.437123 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.437465 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.437549 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.438955 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.438957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.438958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.438986 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.440780 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.441169 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.441170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.441170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.441246 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.441459 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.441460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.441460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.441611 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.441720 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.441847 (+ 127us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.441946 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.28.25.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.60.keytab.
Entry for principal oryx/127.28.25.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.60.keytab.
I20260811 11:15:43.478778 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.60:43933
--webserver_interface=127.28.25.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.60.keytab
--principal=oryx/127.28.25.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:36971,127.28.25.61:41245,127.28.25.60:43933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:15:43.570665 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:43.570925 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:43.570953 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:43.572530 3935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:15:43.572588 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:43.572604 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:43.572618 3935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:15:43.572656 3935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:15:43.574954 3935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:36971,127.28.25.61:41245,127.28.25.60:43933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.60.keytab
--principal=oryx/127.28.25.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.60:43933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.25.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:43.575296 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:43.575630 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:43.581496 3944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:43.581542 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.581692 3935 server_base.cc:1034] running on GCE node
W20260811 11:15:43.581542 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.582134 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:43.582469 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:43.583664 3935 hybrid_clock.cc:648] HybridClock initialized: now 1786446943583636 us; error 48 us; skew 500 ppm
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:43.587877 3935 init.cc:377] Logged in from keytab as oryx/127.28.25.60@KRBTEST.COM (short username oryx)
I20260811 11:15:43.588490 3935 webserver.cc:460] Webserver started at http://127.28.25.60:41927/ using document root <none> and password file <none>
I20260811 11:15:43.588753 3935 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:43.588835 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:43.589016 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:15:43.590437 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data/instance:
uuid: "e90bb810d2c441c286193ff7e79d1075"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "badbf484a9debc4d5dd52ec943c0e34a"
server_key_iv: "c9bf401a596b87c053c58096bb524c72"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.590883 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal/instance:
uuid: "e90bb810d2c441c286193ff7e79d1075"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "badbf484a9debc4d5dd52ec943c0e34a"
server_key_iv: "c9bf401a596b87c053c58096bb524c72"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.592852 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:43.594087 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.594352 3935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:43.594472 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal
uuid: "e90bb810d2c441c286193ff7e79d1075"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "badbf484a9debc4d5dd52ec943c0e34a"
server_key_iv: "c9bf401a596b87c053c58096bb524c72"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.594561 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:43.626322 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:43.629091 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:43.629336 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:43.637682 3935 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.60:43933
I20260811 11:15:43.637786 4012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.60:43933 every 8 connection(s)
I20260811 11:15:43.638218 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
I20260811 11:15:43.639722 4013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:43.641808 4013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:15:43.646636 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 3935
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.60@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.25.60@KRBTEST.COM
I20260811 11:15:43.655197 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.642795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.28.25.62:36971)
0811 11:15:43.642875 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.642878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.643185 (+ 307us) server_negotiation.cc:407] Connection header received
0811 11:15:43.643992 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.643994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.644008 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.644080 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.645279 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.645789 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.647498 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.647837 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.647942 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.651366 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.651368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.651368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.651392 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.653199 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.653684 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.653685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.653686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.653757 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.654166 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.654168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.654168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.654335 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.654409 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.654882 (+ 473us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.655062 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:15:43.655390 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.642633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:56048)
0811 11:15:43.642997 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.643044 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.643839 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.644213 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.644218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.644704 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.645154 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.645160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.645904 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.645905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.647350 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.647353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.648137 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.648140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.648395 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.649239 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.649249 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.651114 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.653310 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.653320 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.653322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.653565 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.653846 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.653847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.653848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.654067 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.654437 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.654446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.654664 (+ 218us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.654832 (+ 168us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.655083 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":254,"threads_started":1}
I20260811 11:15:43.656131 4013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:15:43.658566 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.647393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.28.25.60:43933)
0811 11:15:43.647789 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.647797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.647827 (+ 30us) server_negotiation.cc:407] Connection header received
0811 11:15:43.647852 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.647854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.647879 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.647958 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.648948 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.650088 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.650917 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.651467 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.651542 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.654443 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.654449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.654450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.654458 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.657125 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.657559 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.657562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.657562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.657620 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.657898 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.657899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.657899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.658056 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.658242 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.658347 (+ 105us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.658432 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":162,"threads_started":1}
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.60@KRBTEST.COM for oryx/127.28.25.61@KRBTEST.COM
I20260811 11:15:43.663826 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.656420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41245 (local address 127.0.0.1:40220)
0811 11:15:43.656475 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.656486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.656525 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.656712 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.656713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.657014 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.657104 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.657106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.658133 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.658134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.658922 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.658925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.659514 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.659514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.659570 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.659999 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.660003 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.661239 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.662879 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.662880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.662880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.663127 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.663302 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.663303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.663303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.663357 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.663562 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.663562 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.663603 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.663662 (+ 59us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.663752 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 11:15:43.663900 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.656515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.28.25.61:41245)
0811 11:15:43.656584 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.656586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.656594 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:15:43.656624 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.656625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.656644 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.656709 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.657216 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.658040 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.659087 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.659432 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.659496 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.661351 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.661353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.661353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.661361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.662791 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.663190 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.663191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.663191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.663244 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.663409 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.663409 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.663410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.663513 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.663562 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.663681 (+ 119us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.663757 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 11:15:43.664376 4013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.60" port: 43933 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.60@KRBTEST.COM for oryx/127.28.25.60@KRBTEST.COM
I20260811 11:15:43.671519 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.664722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:43933 (local address 127.0.0.1:38970)
0811 11:15:43.664778 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.664790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.664843 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.664949 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.664950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.665072 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.665112 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.665114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.665777 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.665778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.666310 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.666312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.666751 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.666751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.666821 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.667376 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.667380 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.668679 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.670292 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.670333 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.670333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.670523 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.670728 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.670729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.670729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.670787 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.671137 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.671138 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.671185 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.671257 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.671442 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260811 11:15:43.671648 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.664783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.28.25.60:43933)
0811 11:15:43.664825 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.664827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.664836 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:43.664895 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.664896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.664905 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.664944 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.665187 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.665685 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.666436 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.666649 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.666715 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.668781 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.668782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.668782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.668789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.670187 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.670607 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.670608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.670608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.670664 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.670845 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.670846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.670846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.671081 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.671138 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.671319 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.671502 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":21}
I20260811 11:15:43.673519 4013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: Bootstrap starting.
I20260811 11:15:43.674580 4013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:43.675094 4013 log.cc:824] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:43.676324 4013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: No bootstrap required, opened a new log
I20260811 11:15:43.678009 4013 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 43933 } }
I20260811 11:15:43.678179 4013 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:43.678231 4013 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90bb810d2c441c286193ff7e79d1075, State: Initialized, Role: FOLLOWER
I20260811 11:15:43.678387 4013 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 43933 } }
I20260811 11:15:43.678815 4013 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:43.678799 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 43933 } } }
I20260811 11:15:43.679128 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 11:15:43.681208 4030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:15:43.681283 4030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:15:43.684505 3935 master_runner.cc:428] Detected that this master e90bb810d2c441c286193ff7e79d1075 is joining an existing cluster
I20260811 11:15:43.684571 3935 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.25.60:43933
I20260811 11:15:43.694516 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.686372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.28.25.60:43933)
0811 11:15:43.686426 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.686428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.686578 (+ 150us) server_negotiation.cc:407] Connection header received
0811 11:15:43.686640 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.686640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.686650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.686694 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.687422 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.688169 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.689384 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.689662 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.689704 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.691508 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.691509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.691510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.691518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.693147 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.693604 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.693605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.693605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.693671 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.693932 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.693933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.693933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.694069 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.694147 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.694303 (+ 156us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.694401 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":33}
I20260811 11:15:43.694597 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.686345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:43933 (local address 127.0.0.1:38972)
0811 11:15:43.686523 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.686533 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.686592 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.686728 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.686729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.687202 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.687325 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.687327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.688276 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.688277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.689278 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.689281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.689802 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.689803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.689893 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.690842 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.690848 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.691398 (+ 550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.693326 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.693327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.693327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.693521 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.693790 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.693791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.693791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.693856 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.694149 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.694150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.694199 (+ 49us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.694280 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.694457 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":109,"thread_start_us":71,"threads_started":1}
I20260811 11:15:43.710383 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 3261
I20260811 11:15:43.719825 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.686518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.28.25.61:41245)
0811 11:15:43.686573 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.686575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.686583 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:15:43.686610 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.686611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.686622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.686671 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.687444 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.688201 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.689403 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.689714 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.689755 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.715539 (+ 25784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.715542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.715542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.715551 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.718109 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.718672 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.718674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.718675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.718743 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.719013 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.719013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.719014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.719213 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.719267 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.719537 (+ 270us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.719667 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 11:15:43.720731 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.686069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41245 (local address 127.0.0.1:40228)
0811 11:15:43.686277 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.686290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.686364 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.686734 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.686735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.687198 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.687325 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.687327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.688301 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.688302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.689288 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.689289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.689828 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.689829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.689884 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.714839 (+ 24955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.714847 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.715382 (+ 535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.718239 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.718240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.718241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.718558 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.718829 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.718830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.718830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.718900 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.719337 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.719338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.719416 (+ 78us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.720348 (+ 932us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.720535 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":33,"thread_start_us":101,"threads_started":1}
I20260811 11:15:43.721750 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.686151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.28.25.62:36971)
0811 11:15:43.686206 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.686208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.686504 (+ 296us) server_negotiation.cc:407] Connection header received
0811 11:15:43.687018 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.687019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.687032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.687066 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.687555 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.688334 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.689469 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.689791 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.689837 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.718595 (+ 28758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.718597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.718598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.718606 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.720406 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.720838 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.720839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.720840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.720900 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.721098 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.721098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.721099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.721251 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.721295 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.721558 (+ 263us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.721648 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 11:15:43.722121 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.686076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:56050)
0811 11:15:43.686448 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.686457 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.686526 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.687141 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.687142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.687380 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.687455 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.687457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.688441 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.688442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.689359 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.689362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.689925 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.689926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.689995 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.717955 (+ 27960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.717961 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.718455 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.720523 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.720524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.720525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.720746 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.720971 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.720972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.720972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.721029 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.721373 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.721374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.721436 (+ 62us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.721881 (+ 445us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.722011 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"spinlock_wait_cycles":11648,"thread_start_us":111,"threads_started":1}
I20260811 11:15:43.723501 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:39279
--local_ip_for_outbound_sockets=127.28.25.1
--tserver_master_addrs=127.28.25.62:36971
--webserver_port=41965
--webserver_interface=127.28.25.1
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab
--principal=oryx/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.25.62:36971,127.28.25.61:41245
--tserver_master_addrs=127.28.25.62:36971,127.28.25.61:41245,127.28.25.60:43933 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446942, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.62@KRBTEST.COM for oryx/127.28.25.60@KRBTEST.COM
I20260811 11:15:43.733979 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.724971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:43933 (local address 127.0.0.1:38978)
0811 11:15:43.725031 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.725043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.725083 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.725245 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.725246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.725463 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.725531 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.725534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.726545 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.726548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.727446 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.727452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.727975 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.727977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.728064 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.728596 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.728602 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.730422 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.732677 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.732678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.732679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.732932 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.733183 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.733184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.733184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.733264 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.733604 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.733605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.733662 (+ 57us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.733741 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.733869 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 11:15:43.734027 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.725035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.28.25.60:43933)
0811 11:15:43.725078 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.725080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.725091 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:15:43.725156 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.725157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.725168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.725241 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.725616 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.726424 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.727554 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.727875 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.727953 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.730536 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.730539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.730539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.730551 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.732560 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.733028 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.733030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.733030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.733102 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.733341 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.733342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.733343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.733522 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.733612 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.733793 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.733892 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:15:43.735311 3192 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.25.60:43933: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e90bb810d2c441c286193ff7e79d1075"
member_type: NON_VOTER
last_known_addr {
host: "127.28.25.60"
port: 43933
}
attrs {
promote: true
}
}
dest_uuid: "5d9f13a3038847b7a4c0a65ecae990c4"
cas_config_opid_index: 5
I20260811 11:15:43.735606 3192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:15:43.736156 3833 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e90bb810d2c441c286193ff7e79d1075 (127.28.25.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.737187 3833 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.25.60:43933
I20260811 11:15:43.737290 4042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.737385 4042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:43.738384 3984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:43.739638 3984 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 1 FOLLOWER]: Refusing update from remote peer 5d9f13a3038847b7a4c0a65ecae990c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 11:15:43.739967 4042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:43.741603 3984 raft_consensus.cc:858] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 43933 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
W20260811 11:15:43.741672 3984 raft_consensus.cc:1565] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 43933 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:15:43.742206 3984 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6dbb9aa428404b838f3036368e7f4b77 (127.28.25.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } }
I20260811 11:15:43.742403 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.736337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41245 (local address 127.0.0.1:40232)
0811 11:15:43.737941 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.737954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.738037 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.738290 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.738291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.738293 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:43.738343 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.738345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.739474 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.739475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.741150 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.741153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.742114 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.742115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.742185 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.742214 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.742278 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.742324 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1559}
I20260811 11:15:43.742676 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.736459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.28.25.61:41245)
0811 11:15:43.736509 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.736511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.738003 (+ 1492us) server_negotiation.cc:407] Connection header received
0811 11:15:43.738117 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.738119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.738147 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.738192 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:43.738422 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.739275 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.741275 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.741911 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.741964 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.742047 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.742554 (+ 507us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.742605 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:15:43.742894 3888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 LEARNER]: Refusing update from remote peer 5d9f13a3038847b7a4c0a65ecae990c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 11:15:43.743222 3935 master.cc:537] Master@127.28.25.60:43933 shutting down...
I20260811 11:15:43.743220 4042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260811 11:15:43.744326 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9f13a3038847b7a4c0a65ecae990c4. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } }
I20260811 11:15:43.744477 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260811 11:15:43.744516 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } }
I20260811 11:15:43.744552 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260811 11:15:43.744585 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } }
I20260811 11:15:43.744620 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260811 11:15:43.745493 3888 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e90bb810d2c441c286193ff7e79d1075 (127.28.25.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.746009 4043 mvcc.cc:205] Tried to move back new op lower bound from 7317286676473192448 to 7317286676373286912. Current Snapshot: MvccSnapshot[applied={T|T < 7317286676382511104}]
I20260811 11:15:43.746690 4052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.746893 4052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: This master's current role is: LEARNER
I20260811 11:15:43.747053 3919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9f13a3038847b7a4c0a65ecae990c4. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.747153 3919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: This master's current role is: LEARNER
I20260811 11:15:43.748653 3834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: NON_VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: true } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.748759 3834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:43.748885 3935 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260811 11:15:43.749002 3935 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260811 11:15:43.749037 3935 tablet_replica.cc:330] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: stopping tablet replica
W20260811 11:15:43.749379 3177 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 -> Peer e90bb810d2c441c286193ff7e79d1075 (127.28.25.60:43933): Couldn't send request to peer e90bb810d2c441c286193ff7e79d1075. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 11:15:43.750774 3833 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4: attempting to promote NON_VOTER 6dbb9aa428404b838f3036368e7f4b77 to VOTER
I20260811 11:15:43.751042 3833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:15:43.751673 3888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 LEARNER]: Refusing update from remote peer 5d9f13a3038847b7a4c0a65ecae990c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260811 11:15:43.751993 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.751709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:43933 (local address 127.0.0.1:38984)
0811 11:15:43.751905 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.751961 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:43933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260811 11:15:43.752385 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.752303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:43933 (local address 127.0.0.1:38994)
0811 11:15:43.752340 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.752362 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:43933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260811 11:15:43.752533 3177 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 -> Peer e90bb810d2c441c286193ff7e79d1075 (127.28.25.60:43933): Couldn't send request to peer e90bb810d2c441c286193ff7e79d1075. Status: Network error: Client connection negotiation failed: client connection to 127.28.25.60:43933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 11:15:43.752665 4042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260811 11:15:43.753352 3834 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6dbb9aa428404b838f3036368e7f4b77 (127.28.25.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.754343 3834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.754415 3834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:43.754776 3833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.754853 3833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:15:43.755264 3888 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6dbb9aa428404b838f3036368e7f4b77 (127.28.25.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.756269 3919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.756350 3919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:15:43.763883 3935 master.cc:552] Master@127.28.25.60:43933 shutdown complete.
I20260811 11:15:43.763954 3935 master_runner.cc:305] Clearing existing system tablet
I20260811 11:15:43.764094 3935 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:43.764146 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:43.764969 3935 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:43.765628 4055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.765755 3935 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:43.765808 3935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260811 11:15:43.765831 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal
uuid: "e90bb810d2c441c286193ff7e79d1075"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "badbf484a9debc4d5dd52ec943c0e34a"
server_key_iv: "c9bf401a596b87c053c58096bb524c72"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.766626 3935 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:43.768648 3935 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 11:15:43.768733 3935 log.cc:1196] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260811 11:15:43.769069 3935 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: Deleting consensus metadata
I20260811 11:15:43.769215 3935 master_runner.cc:315] Copying system tablet from 127.28.25.62:36971
I20260811 11:15:43.769963 3935 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: tablet copy: Beginning tablet copy session from remote peer at address 127.28.25.62:36971
I20260811 11:15:43.778393 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.770339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:56052)
0811 11:15:43.770474 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.770486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.770560 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.770735 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.770736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.770949 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.771138 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.771141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.772169 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.772170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.773127 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.773130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.773618 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.773619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.773692 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.774261 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.774266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.774699 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.777152 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.777154 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.777154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.777419 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.777655 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.777656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.777656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.777718 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.778020 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.778021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.778085 (+ 64us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.778149 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.778270 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260811 11:15:43.778709 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.770386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.28.25.62:36971)
0811 11:15:43.770436 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.770438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.770559 (+ 121us) server_negotiation.cc:407] Connection header received
0811 11:15:43.770620 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.770621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.770633 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.770715 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.771229 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.772040 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.773230 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.773523 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.773602 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.775092 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.775093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.775093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.775102 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.777041 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.777514 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.777515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.777516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.777583 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.777786 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.777787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.777787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.777929 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.777992 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.778539 (+ 547us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.778624 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 11:15:43.779017 3214 tablet_copy_service.cc:140] P 5d9f13a3038847b7a4c0a65ecae990c4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e90bb810d2c441c286193ff7e79d1075 ({username='oryx', principal='oryx/127.28.25.60@KRBTEST.COM'} at 127.0.0.1:56052)
I20260811 11:15:43.779081 3214 tablet_copy_service.cc:161] P 5d9f13a3038847b7a4c0a65ecae990c4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e90bb810d2c441c286193ff7e79d1075 at {username='oryx', principal='oryx/127.28.25.60@KRBTEST.COM'} at 127.0.0.1:56052: session id = e90bb810d2c441c286193ff7e79d1075-00000000000000000000000000000000
I20260811 11:15:43.779719 3214 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 11:15:43.780328 3935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 11:15:43.781867 3935 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: tablet copy: Starting download of 0 data blocks...
I20260811 11:15:43.782079 3935 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: tablet copy: Starting download of 1 WAL segments...
I20260811 11:15:43.783053 3935 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 11:15:43.784140 3214 tablet_copy_service.cc:342] P 5d9f13a3038847b7a4c0a65ecae990c4: Request end of tablet copy session e90bb810d2c441c286193ff7e79d1075-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.25.60@KRBTEST.COM'} at 127.0.0.1:56052
I20260811 11:15:43.784197 3214 tablet_copy_service.cc:433] P 5d9f13a3038847b7a4c0a65ecae990c4: ending tablet copy session e90bb810d2c441c286193ff7e79d1075-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e90bb810d2c441c286193ff7e79d1075
I20260811 11:15:43.787613 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:43.788856 4064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:43.789021 4065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.789649 3935 server_base.cc:1034] running on GCE node
W20260811 11:15:43.791117 4067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.791328 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:43.791512 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:43.792717 3935 hybrid_clock.cc:648] HybridClock initialized: now 1786446943792707 us; error 26 us; skew 500 ppm
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:43.794361 3935 init.cc:377] Logged in from keytab as oryx/127.28.25.60@KRBTEST.COM (short username oryx)
I20260811 11:15:43.794678 3935 webserver.cc:460] Webserver started at http://127.28.25.60:33561/ using document root <none> and password file <none>
I20260811 11:15:43.794819 3935 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:43.794858 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:43.795729 3935 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:43.796415 4076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.796599 3935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.796656 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal
uuid: "e90bb810d2c441c286193ff7e79d1075"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "badbf484a9debc4d5dd52ec943c0e34a"
server_key_iv: "c9bf401a596b87c053c58096bb524c72"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.796720 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:43.809018 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:43.811115 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:43.814610 3935 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.60:43933
I20260811 11:15:43.815120 4137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.60:43933 every 8 connection(s)
I20260811 11:15:43.815236 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
I20260811 11:15:43.816596 4138 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:15:43.817162 4138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: Bootstrap starting.
I20260811 11:15:43.819947 4138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:15:43.820266 4138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: Bootstrap complete.
I20260811 11:15:43.820533 4138 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:15:43.820642 4138 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e90bb810d2c441c286193ff7e79d1075, State: Initialized, Role: LEARNER
I20260811 11:15:43.820691 4138 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:15:43.820967 4138 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:15:43.820964 4143 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } } }
I20260811 11:15:43.821229 4143 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075 [sys.catalog]: This master's current role is: LEARNER
I20260811 11:15:43.822777 4155 catalog_manager.cc:1259] Loaded cluster ID: 5e2eb366ca3d4a04a6b4be00857d13af
I20260811 11:15:43.823611 4155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: loading cluster ID for follower catalog manager: success
I20260811 11:15:43.824769 4155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: acquiring CA information for follower catalog manager: success
I20260811 11:15:43.825085 4155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e90bb810d2c441c286193ff7e79d1075: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 11:15:43.838400 4039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:43.838641 4039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:43.838670 4039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:43.840140 4039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:43.840215 4039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:43.840247 4039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:15:43.842070 4039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab
--principal=oryx/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:39279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=41965
--tserver_master_addrs=127.28.25.62:36971,127.28.25.61:41245,127.28.25.60:43933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:43.842398 4039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:43.842692 4039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:43.849553 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:43.849521 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:43.849852 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.849825 4039 server_base.cc:1034] running on GCE node
I20260811 11:15:43.850193 4039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:43.850524 4039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:43.851789 4039 hybrid_clock.cc:648] HybridClock initialized: now 1786446943851742 us; error 58 us; skew 500 ppm
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:43.854939 4039 init.cc:377] Logged in from keytab as oryx/127.28.25.1@KRBTEST.COM (short username oryx)
I20260811 11:15:43.855638 4039 webserver.cc:460] Webserver started at http://127.28.25.1:41965/ using document root <none> and password file <none>
I20260811 11:15:43.855935 4039 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:43.856002 4039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:43.857628 4039 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 11:15:43.858376 4171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.858554 4039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:15:43.858652 4039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "9d3925f34ae44afb9de4bcb773260034"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "0775b7adc704e5ba4b225d73e7c63a68"
server_key_iv: "cc382c6bd05b8312c5839c5abe2f191b"
server_key_version: "encryptionkey@0"
I20260811 11:15:43.859086 4039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:43.867993 4039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:43.870450 4039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:43.870594 4039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:43.870810 4039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:43.871205 4039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:43.871249 4039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.871274 4039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:43.871287 4039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:43.880100 4039 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:39279
I20260811 11:15:43.880220 4300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:39279 every 8 connection(s)
I20260811 11:15:43.880671 4039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:15:43.881597 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 4039
I20260811 11:15:43.881820 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 3412
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.61@KRBTEST.COM
Aug 11 11:15:43 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446943, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.60@KRBTEST.COM
I20260811 11:15:43.892004 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:36035
--local_ip_for_outbound_sockets=127.28.25.2
--tserver_master_addrs=127.28.25.62:36971
--webserver_port=42353
--webserver_interface=127.28.25.2
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab
--principal=oryx/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.25.62:36971,127.28.25.61:41245
--tserver_master_addrs=127.28.25.62:36971,127.28.25.61:41245,127.28.25.60:43933 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:15:43.896430 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.881972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:36771 (local address 127.28.25.62:36971)
0811 11:15:43.882161 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.882164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.882300 (+ 136us) server_negotiation.cc:407] Connection header received
0811 11:15:43.883342 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.883344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.883358 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.883406 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.884658 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.885487 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.887403 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.887744 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.887796 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.891578 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.891580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.891580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.891590 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.894669 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.895401 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.895403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.895403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.895473 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.895680 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.895682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.895682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.895842 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.895898 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.896248 (+ 350us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.896346 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260811 11:15:43.896481 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.884790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:33083 (local address 127.28.25.61:41245)
0811 11:15:43.884847 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.884848 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.885017 (+ 169us) server_negotiation.cc:407] Connection header received
0811 11:15:43.885112 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.885113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.885124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.885156 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.885562 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.886294 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.887351 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.887655 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.887701 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.892132 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.892135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.892135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.892145 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.894252 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.895070 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.895071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.895072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.895145 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.895524 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.895526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.895526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.895714 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.895775 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.896268 (+ 493us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.896368 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 11:15:43.896806 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.881745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:43933 (local address 127.28.25.1:44307)
0811 11:15:43.882115 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.882182 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.883201 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.883569 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.883574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.884211 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.884831 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.884834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.886089 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.886090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.887227 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.887231 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.887787 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.887788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.887970 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.889619 (+ 1649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.889625 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.892321 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.894856 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.894858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.894858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.895175 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.895492 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.895492 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.895493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.895564 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.895919 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.895920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.896077 (+ 157us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.896184 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.896484 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":18,"thread_start_us":126,"threads_started":1}
I20260811 11:15:43.896898 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.881745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:36771)
0811 11:15:43.882194 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.882204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.883201 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.883569 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.883574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.884243 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.884558 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.884567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.885674 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.885675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.887248 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.887252 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.887959 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.887960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.888055 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.889014 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.889019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.891406 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.894772 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.894773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.894774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.895305 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.895549 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.895550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.895551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.895611 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.895936 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.895937 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.896070 (+ 133us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.896597 (+ 527us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.896735 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":45,"spinlock_wait_cycles":9472,"thread_start_us":200,"threads_started":1}
I20260811 11:15:43.897369 4302 heartbeater.cc:347] Connected to a master server at 127.28.25.62:36971
I20260811 11:15:43.897397 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.881809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:44307 (local address 127.28.25.60:43933)
0811 11:15:43.882100 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:43.882103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:43.882382 (+ 279us) server_negotiation.cc:407] Connection header received
0811 11:15:43.883388 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:43.883390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:43.883406 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:43.883477 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:43.884914 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.885989 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.887351 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.887682 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.888266 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.892443 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:43.892445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:43.892446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:43.892455 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:43.894737 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.895287 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.895289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.895289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.895358 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:43.895648 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:43.895649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:43.895649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:43.895805 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:43.896949 (+ 1144us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:43.897023 (+ 74us) server_negotiation.cc:299] Negotiation successful
0811 11:15:43.897188 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":85,"threads_started":1}
I20260811 11:15:43.897477 4302 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:43.897704 4302 heartbeater.cc:511] Master 127.28.25.62:36971 requested a full tablet report, sending...
I20260811 11:15:43.897768 4301 heartbeater.cc:347] Connected to a master server at 127.28.25.60:43933
I20260811 11:15:43.897836 4301 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:43.897925 4301 heartbeater.cc:511] Master 127.28.25.60:43933 requested a full tablet report, sending...
I20260811 11:15:43.898147 3192 ts_manager.cc:194] Re-registered known tserver with Master: 9d3925f34ae44afb9de4bcb773260034 (127.28.25.1:39279)
I20260811 11:15:43.898350 4086 ts_manager.cc:194] Registered new tserver with Master: 9d3925f34ae44afb9de4bcb773260034 (127.28.25.1:39279)
I20260811 11:15:43.898653 3192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:36771
I20260811 11:15:43.903532 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:43.884559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41245 (local address 127.28.25.1:33083)
0811 11:15:43.884963 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:43.884973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:43.885040 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:43.885219 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:43.885220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:43.885407 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:43.885467 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.885468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.886389 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.886390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.887227 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:43.887231 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:43.887787 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:43.887788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:43.887970 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:43.888860 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:43.888873 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:43.891993 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:43.894387 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.894404 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.894408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.894855 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.895243 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:43.895243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:43.895244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:43.895432 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:43.895814 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:43.895823 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:43.896071 (+ 248us) client_negotiation.cc:769] Sending connection context
0811 11:15:43.903063 (+ 6992us) client_negotiation.cc:240] Negotiation successful
0811 11:15:43.903324 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260811 11:15:43.904201 4303 heartbeater.cc:347] Connected to a master server at 127.28.25.61:41245
I20260811 11:15:43.904282 4303 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:43.904418 4303 heartbeater.cc:511] Master 127.28.25.61:41245 requested a full tablet report, sending...
I20260811 11:15:43.904856 3877 ts_manager.cc:194] Registered new tserver with Master: 9d3925f34ae44afb9de4bcb773260034 (127.28.25.1:39279)
W20260811 11:15:43.989701 4310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:43.989974 4310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:43.990039 4310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:43.991295 4310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:43.991349 4310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:43.991379 4310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:15:43.993006 4310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab
--principal=oryx/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:36035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=42353
--tserver_master_addrs=127.28.25.62:36971,127.28.25.61:41245,127.28.25.60:43933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:43.993297 4310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:43.993568 4310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:43.999264 4316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:43.999287 4319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:43.999264 4317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:43.999296 4310 server_base.cc:1034] running on GCE node
I20260811 11:15:43.999698 4310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:43.999975 4310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:44.001196 4310 hybrid_clock.cc:648] HybridClock initialized: now 1786446944001172 us; error 44 us; skew 500 ppm
Aug 11 11:15:44 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446944, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:44.004249 4310 init.cc:377] Logged in from keytab as oryx/127.28.25.2@KRBTEST.COM (short username oryx)
I20260811 11:15:44.004797 4310 webserver.cc:460] Webserver started at http://127.28.25.2:42353/ using document root <none> and password file <none>
I20260811 11:15:44.005069 4310 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:44.005159 4310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:44.006673 4310 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260811 11:15:44.007579 4326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:44.007823 4310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:15:44.007947 4310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "306af20b690a46bf83a8b269cb11cb81"
format_stamp: "Formatted at 2026-08-11 11:15:42 on dist-test-slave-2w3w"
server_key: "3f7f308da1e127698d434c335d62e6c6"
server_key_iv: "c5cddbfe7b51645670512f5f400ff07f"
server_key_version: "encryptionkey@0"
I20260811 11:15:44.008376 4310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:44.016609 4310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:44.019230 4310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:44.019418 4310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:44.019696 4310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:44.020053 4310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:44.020110 4310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:44.020156 4310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:44.020187 4310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:44.027244 4310 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:36035
I20260811 11:15:44.027289 4460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:36035 every 8 connection(s)
I20260811 11:15:44.027673 4310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:15:44.031199 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 4310
I20260811 11:15:44.031412 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 3568
Aug 11 11:15:44 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446944, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.61@KRBTEST.COM
Aug 11 11:15:44 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446944, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
Aug 11 11:15:44 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446944, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.60@KRBTEST.COM
I20260811 11:15:44.042301 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:43521
--local_ip_for_outbound_sockets=127.28.25.3
--tserver_master_addrs=127.28.25.62:36971
--webserver_port=34303
--webserver_interface=127.28.25.3
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab
--principal=oryx/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.25.62:36971,127.28.25.61:41245
--tserver_master_addrs=127.28.25.62:36971,127.28.25.61:41245,127.28.25.60:43933 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:15:44.043807 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.028613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:39833 (local address 127.28.25.60:43933)
0811 11:15:44.028776 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.028778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.030875 (+ 2097us) server_negotiation.cc:407] Connection header received
0811 11:15:44.031044 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.031046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.031059 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.031117 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.033961 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.034827 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.036571 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.036999 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.037056 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.039485 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.039486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.039487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.039497 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.041618 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.042370 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.042372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.042372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.042444 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.042801 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.042802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.042802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.043008 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.043084 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.043544 (+ 460us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.043662 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260811 11:15:44.043934 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.028668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:40595 (local address 127.28.25.62:36971)
0811 11:15:44.028821 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.028823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.030867 (+ 2044us) server_negotiation.cc:407] Connection header received
0811 11:15:44.030908 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.030909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.030921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.031056 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.032155 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.032897 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.034568 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.035016 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.035064 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.038775 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.038776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.038777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.038787 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.041781 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.042390 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.042392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.042392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.042464 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.043118 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.043120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.043120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.043272 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.043329 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.043676 (+ 347us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.043772 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260811 11:15:44.044721 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.028476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:40595)
0811 11:15:44.030625 (+ 2149us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.030636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.030714 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.031117 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.031119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.031972 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:44.032043 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.032046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.033040 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.033041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.034445 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.034448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.035158 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.035159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.035282 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.036102 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:44.036107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:44.038645 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:44.041875 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.041876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.041876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.042273 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.042570 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.042571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.042571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.042726 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.043499 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:44.043500 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:44.043551 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.044344 (+ 793us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.044481 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2118,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260811 11:15:44.044725 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.028476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:43933 (local address 127.28.25.2:39833)
0811 11:15:44.030804 (+ 2328us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.030825 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.030902 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.031166 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.031167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.031554 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:44.033838 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.033842 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.034935 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.034936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.036437 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.036441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.037153 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.037154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.037227 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.037715 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:44.037720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:44.039361 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:44.041744 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.041757 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.041761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.042273 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.042536 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.042537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.042537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.042719 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.043075 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:44.043085 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:44.043338 (+ 253us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.043990 (+ 652us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.044316 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2292,"mutex_wait_us":2065,"thread_start_us":98,"threads_started":1}
I20260811 11:15:44.044824 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.028550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:58915 (local address 127.28.25.61:41245)
0811 11:15:44.028720 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.028723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.028936 (+ 213us) server_negotiation.cc:407] Connection header received
0811 11:15:44.030599 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.030600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.030613 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.030689 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.032007 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.033037 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.034554 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.034927 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.034976 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.039506 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.039508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.039508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.039518 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.041810 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.043507 (+ 1697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.043508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.043508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.043583 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.044133 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.044135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.044135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.044298 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.044365 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.044586 (+ 221us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.044682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260811 11:15:44.045269 4461 heartbeater.cc:347] Connected to a master server at 127.28.25.60:43933
I20260811 11:15:44.045398 4461 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:44.045527 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.028476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41245 (local address 127.28.25.2:58915)
0811 11:15:44.028767 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.028829 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.030415 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.030805 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.030809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.031439 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:44.031887 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.031894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.033153 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.033154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.034449 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.034452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.035108 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.035109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.035280 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.036020 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:44.036031 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:44.039393 (+ 3362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:44.041901 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.041902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.041903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.043388 (+ 1485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.043675 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.043675 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.043676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.043738 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.044405 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:44.044406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:44.044453 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.045268 (+ 815us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.045415 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260811 11:15:44.045637 4461 heartbeater.cc:511] Master 127.28.25.60:43933 requested a full tablet report, sending...
I20260811 11:15:44.046080 4465 heartbeater.cc:347] Connected to a master server at 127.28.25.61:41245
I20260811 11:15:44.046132 4465 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:44.046135 4086 ts_manager.cc:194] Registered new tserver with Master: 306af20b690a46bf83a8b269cb11cb81 (127.28.25.2:36035)
I20260811 11:15:44.046200 4465 heartbeater.cc:511] Master 127.28.25.61:41245 requested a full tablet report, sending...
I20260811 11:15:44.046504 3877 ts_manager.cc:194] Registered new tserver with Master: 306af20b690a46bf83a8b269cb11cb81 (127.28.25.2:36035)
I20260811 11:15:44.046720 4464 heartbeater.cc:347] Connected to a master server at 127.28.25.62:36971
I20260811 11:15:44.046769 4464 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:44.046856 4464 heartbeater.cc:511] Master 127.28.25.62:36971 requested a full tablet report, sending...
I20260811 11:15:44.047140 3192 ts_manager.cc:194] Re-registered known tserver with Master: 306af20b690a46bf83a8b269cb11cb81 (127.28.25.2:36035)
I20260811 11:15:44.047539 3192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:40595
W20260811 11:15:44.140156 4470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:15:44.140462 4470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:15:44.140515 4470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:15:44.141983 4470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:15:44.142076 4470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:15:44.142132 4470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:15:44.144052 4470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab
--principal=oryx/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:43521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=34303
--tserver_master_addrs=127.28.25.62:36971,127.28.25.61:41245,127.28.25.60:43933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:15:44.144392 4470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:15:44.144673 4470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:15:44.150846 4477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:44.151015 4470 server_base.cc:1034] running on GCE node
W20260811 11:15:44.150907 4476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:15:44.150938 4479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:15:44.151368 4470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:15:44.151674 4470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:15:44.152915 4470 hybrid_clock.cc:648] HybridClock initialized: now 1786446944152924 us; error 59 us; skew 500 ppm
Aug 11 11:15:44 dist-test-slave-2w3w krb5kdc[3139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446944, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:15:44.155951 4470 init.cc:377] Logged in from keytab as oryx/127.28.25.3@KRBTEST.COM (short username oryx)
I20260811 11:15:44.156409 4470 webserver.cc:460] Webserver started at http://127.28.25.3:34303/ using document root <none> and password file <none>
I20260811 11:15:44.156636 4470 fs_manager.cc:359] Metadata directory not provided
I20260811 11:15:44.156695 4470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:15:44.158171 4470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:44.159049 4487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:44.159356 4470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:15:44.159443 4470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab"
format_stamp: "Formatted at 2026-08-11 11:15:43 on dist-test-slave-2w3w"
server_key: "f43c7c3284efd9563f5d84970b667d91"
server_key_iv: "e485236f9cfb2743030d0b30a352705c"
server_key_version: "encryptionkey@0"
I20260811 11:15:44.159909 4470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:15:44.177316 4470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:15:44.180121 4470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:15:44.180318 4470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:15:44.180629 4470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:15:44.181035 4470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:15:44.181082 4470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:44.181128 4470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:15:44.181160 4470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:15:44.188006 4470 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:43521
I20260811 11:15:44.188104 4619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:43521 every 8 connection(s)
I20260811 11:15:44.188550 4470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:15:44.191164 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 4470
Aug 11 11:15:44 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446944, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.61@KRBTEST.COM
Aug 11 11:15:44 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446944, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.60@KRBTEST.COM
Aug 11 11:15:44 dist-test-slave-2w3w krb5kdc[3139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446944, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:15:44.203259 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.189430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:43933 (local address 127.28.25.3:45407)
0811 11:15:44.189807 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.189818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.190403 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.191263 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.191264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.191453 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:44.191617 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.191623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.192721 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.192722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.194280 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.194284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.194912 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.194912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.195030 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.196186 (+ 1156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:44.196191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:44.198722 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:44.201134 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.201150 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.201153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.201526 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.201770 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.201771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.201771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.201943 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.202269 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:44.202278 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:44.202493 (+ 215us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.202623 (+ 130us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.202883 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":330,"thread_start_us":67,"threads_started":1}
I20260811 11:15:44.203339 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.189529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:45407 (local address 127.28.25.60:43933)
0811 11:15:44.189587 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.189589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.189905 (+ 316us) server_negotiation.cc:407] Connection header received
0811 11:15:44.190515 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.190517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.190532 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.190600 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.191750 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.192606 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.194401 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.194769 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.194824 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.198865 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.198867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.198867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.198876 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.201028 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.201625 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.201626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.201626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.201694 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.202021 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.202022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.202023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.202174 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.202239 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.203097 (+ 858us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.203199 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 11:15:44.204079 4620 heartbeater.cc:347] Connected to a master server at 127.28.25.60:43933
I20260811 11:15:44.204208 4620 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:44.204479 4620 heartbeater.cc:511] Master 127.28.25.60:43933 requested a full tablet report, sending...
I20260811 11:15:44.205008 4086 ts_manager.cc:194] Registered new tserver with Master: 4d17e5e753e64c4faf59dacb5ec6b4ab (127.28.25.3:43521)
I20260811 11:15:44.205816 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.189594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:37155 (local address 127.28.25.61:41245)
0811 11:15:44.189684 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.189686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.189874 (+ 188us) server_negotiation.cc:407] Connection header received
0811 11:15:44.190720 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.190721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.190733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.190791 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.191749 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.192648 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.194401 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.194765 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.194815 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.199769 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.199771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.199771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.199781 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.203523 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.204630 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.204631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.204631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.204711 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.204976 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.204977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.204977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.205150 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.205200 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.205576 (+ 376us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.205685 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 11:15:44.206384 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.189534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:35037 (local address 127.28.25.62:36971)
0811 11:15:44.189648 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.189650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.189897 (+ 247us) server_negotiation.cc:407] Connection header received
0811 11:15:44.190579 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.190580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.190591 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.190656 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.191740 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.192501 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.194375 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.194690 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.194749 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.199574 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.199575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.199576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.199584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.202286 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.205386 (+ 3100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.205388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.205389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.205459 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.205687 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.205688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.205688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.205889 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.205953 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.206180 (+ 227us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.206261 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 11:15:44.206579 4631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.192380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.28.25.62:36971)
0811 11:15:44.193515 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.193517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.193688 (+ 171us) server_negotiation.cc:407] Connection header received
0811 11:15:44.193710 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.193711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.193723 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.193785 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.194670 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.195694 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.197050 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.197355 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.197425 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.199064 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.199065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.199066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.199074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.205012 (+ 5938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.205433 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.205434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.205434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.205484 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.205687 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.205688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.205688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.205891 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.205991 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.206405 (+ 414us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.206495 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1097,"thread_start_us":77,"threads_started":1}
I20260811 11:15:44.206992 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.189430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:35037)
0811 11:15:44.189768 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.189777 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.190382 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.190725 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.190730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.191271 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:44.191617 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.191624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.192650 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.192651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.194259 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.194262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.194845 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.194846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.195028 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.196277 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:44.196282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:44.199266 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:44.203020 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.203021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.203022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.203584 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.205541 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.205542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.205543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.205607 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.205990 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:44.205991 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:44.206055 (+ 64us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.206510 (+ 455us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.206630 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":148,"thread_start_us":82,"threads_started":1}
I20260811 11:15:44.207309 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.189430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41245 (local address 127.28.25.3:37155)
0811 11:15:44.189717 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.189765 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.190382 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.190814 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.190815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.191342 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:15:44.191617 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.191623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.192749 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.192749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.194286 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.194293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.194910 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.194911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.195038 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.195831 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:15:44.195843 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:15:44.199646 (+ 3803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:15:44.204049 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.204050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.204051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.204455 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.204812 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:15:44.204813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:15:44.204814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:15:44.204887 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:15:44.205347 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:15:44.205348 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:15:44.205403 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.206196 (+ 793us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.206356 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":136,"thread_start_us":97,"threads_started":1}
I20260811 11:15:44.207324 4621 heartbeater.cc:347] Connected to a master server at 127.28.25.62:36971
I20260811 11:15:44.207598 4621 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:44.207690 4621 heartbeater.cc:511] Master 127.28.25.62:36971 requested a full tablet report, sending...
I20260811 11:15:44.207902 4622 heartbeater.cc:347] Connected to a master server at 127.28.25.61:41245
I20260811 11:15:44.207953 4622 heartbeater.cc:464] Registering TS with master...
I20260811 11:15:44.207942 3192 ts_manager.cc:194] Re-registered known tserver with Master: 4d17e5e753e64c4faf59dacb5ec6b4ab (127.28.25.3:43521)
I20260811 11:15:44.208019 4622 heartbeater.cc:511] Master 127.28.25.61:41245 requested a full tablet report, sending...
I20260811 11:15:44.208153 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.192810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.28.25.60:43933)
0811 11:15:44.193857 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.193860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.195386 (+ 1526us) server_negotiation.cc:407] Connection header received
0811 11:15:44.195455 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.195455 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.195471 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.195547 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.196917 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.197645 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.198748 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.199182 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.199248 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.200787 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.200788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.200789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.200797 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.202653 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.203485 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.203486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.203486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.203561 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.203814 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.203815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.203815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.203965 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.204066 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.207682 (+ 3616us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.207810 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":60,"threads_started":1}
I20260811 11:15:44.208331 3877 ts_manager.cc:194] Registered new tserver with Master: 4d17e5e753e64c4faf59dacb5ec6b4ab (127.28.25.3:43521)
I20260811 11:15:44.208392 3192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:35037
I20260811 11:15:44.209702 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.193330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.28.25.61:41245)
0811 11:15:44.196890 (+ 3560us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.196893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.196902 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:44.196928 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.196929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.196943 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.197012 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.199394 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.200679 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.201890 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.202234 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.202303 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.205206 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.205207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.205207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.205216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.207308 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.208898 (+ 1590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.208900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.208900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.208972 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.209183 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.209184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.209184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.209322 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.209435 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.209517 (+ 82us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.209611 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3532,"thread_start_us":59,"threads_started":1}
I20260811 11:15:44.208673 3191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:15:44.210881 3191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:15:44.219832 4647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.215819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.28.25.1:39279)
0811 11:15:44.216035 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.216040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.216051 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:15:44.216097 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.216099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.216117 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.216207 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:44.216374 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.217236 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.218632 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.219425 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.219506 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.219590 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.219688 (+ 98us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.219733 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":128,"threads_started":1}
I20260811 11:15:44.219831 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.215445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:39279 (local address 127.0.0.1:60392)
0811 11:15:44.215631 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.215646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.215694 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.216219 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.216220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.216221 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:44.216275 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.216277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.217339 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.217340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.218514 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.218517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.219526 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.219527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.219598 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.219602 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.219673 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.219724 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260811 11:15:44.220453 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.215543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.28.25.3:43521)
0811 11:15:44.216010 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.216016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.216162 (+ 146us) server_negotiation.cc:407] Connection header received
0811 11:15:44.216202 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.216204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.216218 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.216287 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:44.216416 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.217516 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.218754 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.219466 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.219570 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.219637 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.219894 (+ 257us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.220373 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":90,"threads_started":1}
I20260811 11:15:44.220692 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.215456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:43521 (local address 127.0.0.1:56552)
0811 11:15:44.215856 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.216041 (+ 185us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.216105 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.216304 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.216305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.216306 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:44.216356 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.216357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.217856 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.217887 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.218660 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.218662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.219676 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.219678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.219735 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.219739 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.219791 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.219826 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":114,"threads_started":1}
I20260811 11:15:44.220682 4228 tablet_service.cc:1467] Processing CreateTablet for tablet 2b219f5d5f1a4d70b8d96eeabecd7db9 (DEFAULT_TABLE table=test-table [id=30b603ac4a9a43d2823ccc44f96b4963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:44.221158 4228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b219f5d5f1a4d70b8d96eeabecd7db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:44.223429 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.215518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:36035 (local address 127.0.0.1:53882)
0811 11:15:44.216066 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.216079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.216142 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.216497 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.216498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.216501 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:44.216534 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.216535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.217753 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.217754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.218960 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.218963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.220016 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.220017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.223190 (+ 3173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.223202 (+ 12us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.223251 (+ 49us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.223318 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":103,"threads_started":1}
I20260811 11:15:44.223845 4646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.215834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.28.25.2:36035)
0811 11:15:44.216339 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.216343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.216352 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:15:44.216402 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.216404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.216420 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.216501 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:44.216591 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.217664 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.219063 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.219802 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.220188 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.223652 (+ 3464us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.223687 (+ 35us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.223757 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":119,"threads_started":1}
I20260811 11:15:44.224051 4541 tablet_service.cc:1467] Processing CreateTablet for tablet 2b219f5d5f1a4d70b8d96eeabecd7db9 (DEFAULT_TABLE table=test-table [id=30b603ac4a9a43d2823ccc44f96b4963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:44.224512 4541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b219f5d5f1a4d70b8d96eeabecd7db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:44.224617 4384 tablet_service.cc:1467] Processing CreateTablet for tablet 2b219f5d5f1a4d70b8d96eeabecd7db9 (DEFAULT_TABLE table=test-table [id=30b603ac4a9a43d2823ccc44f96b4963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:15:44.224963 4384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b219f5d5f1a4d70b8d96eeabecd7db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:15:44.225891 4648 tablet_bootstrap.cc:492] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: Bootstrap starting.
I20260811 11:15:44.226550 4648 tablet_bootstrap.cc:654] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:44.226895 4648 log.cc:824] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:44.227749 4648 tablet_bootstrap.cc:492] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: No bootstrap required, opened a new log
I20260811 11:15:44.227816 4648 ts_tablet_manager.cc:1397] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:15:44.229003 4652 tablet_bootstrap.cc:492] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: Bootstrap starting.
I20260811 11:15:44.229333 4651 tablet_bootstrap.cc:492] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: Bootstrap starting.
I20260811 11:15:44.229846 4652 tablet_bootstrap.cc:654] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:44.229717 4648 raft_consensus.cc:348] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 } } peers { permanent_uuid: "9d3925f34ae44afb9de4bcb773260034" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39279 } } peers { permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 } }
I20260811 11:15:44.229898 4648 raft_consensus.cc:374] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:44.229938 4648 raft_consensus.cc:729] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3925f34ae44afb9de4bcb773260034, State: Initialized, Role: FOLLOWER
I20260811 11:15:44.230094 4651 tablet_bootstrap.cc:654] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: Neither blocks nor log segments found. Creating new log.
I20260811 11:15:44.230118 4648 consensus_queue.cc:260] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 } } peers { permanent_uuid: "9d3925f34ae44afb9de4bcb773260034" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39279 } } peers { permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 } }
I20260811 11:15:44.230206 4652 log.cc:824] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:44.230475 4648 ts_tablet_manager.cc:1428] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:15:44.230530 4651 log.cc:824] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: Log is configured to *not* fsync() on all Append() calls
I20260811 11:15:44.230590 4302 heartbeater.cc:503] Master 127.28.25.62:36971 was elected leader, sending a full tablet report...
I20260811 11:15:44.232234 4651 tablet_bootstrap.cc:492] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: No bootstrap required, opened a new log
I20260811 11:15:44.232335 4651 ts_tablet_manager.cc:1397] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 11:15:44.232383 4652 tablet_bootstrap.cc:492] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: No bootstrap required, opened a new log
I20260811 11:15:44.232494 4652 ts_tablet_manager.cc:1397] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 11:15:44.233947 4651 raft_consensus.cc:348] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 } } peers { permanent_uuid: "9d3925f34ae44afb9de4bcb773260034" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39279 } } peers { permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 } }
I20260811 11:15:44.234138 4651 raft_consensus.cc:374] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:44.234189 4651 raft_consensus.cc:729] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d17e5e753e64c4faf59dacb5ec6b4ab, State: Initialized, Role: FOLLOWER
I20260811 11:15:44.234341 4651 consensus_queue.cc:260] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 } } peers { permanent_uuid: "9d3925f34ae44afb9de4bcb773260034" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39279 } } peers { permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 } }
I20260811 11:15:44.234671 4651 ts_tablet_manager.cc:1428] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 11:15:44.234792 4621 heartbeater.cc:503] Master 127.28.25.62:36971 was elected leader, sending a full tablet report...
I20260811 11:15:44.234889 4652 raft_consensus.cc:348] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 } } peers { permanent_uuid: "9d3925f34ae44afb9de4bcb773260034" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39279 } } peers { permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 } }
I20260811 11:15:44.235119 4652 raft_consensus.cc:374] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:15:44.235169 4652 raft_consensus.cc:729] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306af20b690a46bf83a8b269cb11cb81, State: Initialized, Role: FOLLOWER
I20260811 11:15:44.235328 4652 consensus_queue.cc:260] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 } } peers { permanent_uuid: "9d3925f34ae44afb9de4bcb773260034" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39279 } } peers { permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 } }
I20260811 11:15:44.235678 4652 ts_tablet_manager.cc:1428] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260811 11:15:44.236001 4464 heartbeater.cc:503] Master 127.28.25.62:36971 was elected leader, sending a full tablet report...
I20260811 11:15:44.251318 4653 raft_consensus.cc:482] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:15:44.251456 4653 raft_consensus.cc:504] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 } } peers { permanent_uuid: "9d3925f34ae44afb9de4bcb773260034" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39279 } } peers { permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 } }
I20260811 11:15:44.251930 4653 leader_election.cc:288] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d17e5e753e64c4faf59dacb5ec6b4ab (127.28.25.3:43521), 306af20b690a46bf83a8b269cb11cb81 (127.28.25.2:36035)
I20260811 11:15:44.254843 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.252160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:40585 (local address 127.28.25.3:43521)
0811 11:15:44.252299 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.252301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.252313 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:15:44.252341 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.252342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.252355 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.252406 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:44.252593 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.253068 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.254095 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.254462 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.254546 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.254557 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.254729 (+ 172us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.254769 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260811 11:15:44.254843 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.252079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:43521 (local address 127.28.25.1:40585)
0811 11:15:44.252181 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.252194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.252271 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.252423 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.252424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.252425 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:44.252513 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.252515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.253174 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.253175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.253991 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.253993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.254546 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.254547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.254633 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.254636 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.254698 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.254759 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":26}
I20260811 11:15:44.255273 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.252124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:36035 (local address 127.28.25.1:41651)
0811 11:15:44.252181 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:44.252194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:15:44.252238 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:15:44.252422 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:15:44.252423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:15:44.252425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:15:44.252522 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.252524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.253157 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.253158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.254098 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:15:44.254101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.254849 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.254850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:15:44.255009 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.255014 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:15:44.255082 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 11:15:44.255136 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":82}
I20260811 11:15:44.255410 4646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.252209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:41651 (local address 127.28.25.2:36035)
0811 11:15:44.252263 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.252265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.252277 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:15:44.252307 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.252308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.252321 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.252404 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:15:44.252593 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.253068 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.254221 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.254678 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.254797 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.254809 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.255282 (+ 473us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.255341 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 11:15:44.255422 4563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b219f5d5f1a4d70b8d96eeabecd7db9" candidate_uuid: "9d3925f34ae44afb9de4bcb773260034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" is_pre_election: true
I20260811 11:15:44.255631 4563 raft_consensus.cc:2454] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d3925f34ae44afb9de4bcb773260034 in term 0.
I20260811 11:15:44.255720 4405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b219f5d5f1a4d70b8d96eeabecd7db9" candidate_uuid: "9d3925f34ae44afb9de4bcb773260034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306af20b690a46bf83a8b269cb11cb81" is_pre_election: true
I20260811 11:15:44.255898 4405 raft_consensus.cc:2454] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d3925f34ae44afb9de4bcb773260034 in term 0.
I20260811 11:15:44.256338 4172 leader_election.cc:302] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d17e5e753e64c4faf59dacb5ec6b4ab, 9d3925f34ae44afb9de4bcb773260034; no voters:
I20260811 11:15:44.256611 4653 raft_consensus.cc:2781] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:15:44.256717 4653 raft_consensus.cc:482] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:15:44.256758 4653 raft_consensus.cc:3037] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:44.257573 4653 raft_consensus.cc:504] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 } } peers { permanent_uuid: "9d3925f34ae44afb9de4bcb773260034" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39279 } } peers { permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 } }
I20260811 11:15:44.257756 4653 leader_election.cc:288] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [CANDIDATE]: Term 1 election: Requested vote from peers 4d17e5e753e64c4faf59dacb5ec6b4ab (127.28.25.3:43521), 306af20b690a46bf83a8b269cb11cb81 (127.28.25.2:36035)
I20260811 11:15:44.257987 4405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b219f5d5f1a4d70b8d96eeabecd7db9" candidate_uuid: "9d3925f34ae44afb9de4bcb773260034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306af20b690a46bf83a8b269cb11cb81"
I20260811 11:15:44.258092 4405 raft_consensus.cc:3037] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:44.258919 4405 raft_consensus.cc:2454] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d3925f34ae44afb9de4bcb773260034 in term 1.
I20260811 11:15:44.259097 4563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b219f5d5f1a4d70b8d96eeabecd7db9" candidate_uuid: "9d3925f34ae44afb9de4bcb773260034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab"
I20260811 11:15:44.259213 4563 raft_consensus.cc:3037] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:15:44.259209 4174 leader_election.cc:302] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306af20b690a46bf83a8b269cb11cb81, 9d3925f34ae44afb9de4bcb773260034; no voters:
I20260811 11:15:44.259384 4653 raft_consensus.cc:2781] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:15:44.259559 4653 raft_consensus.cc:686] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 1 LEADER]: Becoming Leader. State: Replica: 9d3925f34ae44afb9de4bcb773260034, State: Running, Role: LEADER
I20260811 11:15:44.259699 4653 consensus_queue.cc:237] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 } } peers { permanent_uuid: "9d3925f34ae44afb9de4bcb773260034" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39279 } } peers { permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 } }
I20260811 11:15:44.260237 4563 raft_consensus.cc:2454] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d3925f34ae44afb9de4bcb773260034 in term 1.
I20260811 11:15:44.260636 3192 catalog_manager.cc:5495] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d3925f34ae44afb9de4bcb773260034 (127.28.25.1). New cstate: current_term: 1 leader_uuid: "9d3925f34ae44afb9de4bcb773260034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d3925f34ae44afb9de4bcb773260034" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 39279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 } health_report { overall_health: UNKNOWN } } }
W20260811 11:15:44.278498 4466 tablet.cc:2369] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:15:44.279212 4647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.276539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.28.25.1:39279)
0811 11:15:44.276606 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.276608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.276618 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:15:44.276665 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.276665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.276689 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.276766 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:15:44.276957 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.277509 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.278291 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.278504 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.278566 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.278802 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:15:44.278913 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:15:44.279059 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.279074 (+ 15us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.279097 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:15:44.281574 4405 raft_consensus.cc:1261] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81 [term 1 FOLLOWER]: Refusing update from remote peer 9d3925f34ae44afb9de4bcb773260034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:44.281575 4563 raft_consensus.cc:1261] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab [term 1 FOLLOWER]: Refusing update from remote peer 9d3925f34ae44afb9de4bcb773260034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:15:44.282001 4659 consensus_queue.cc:1035] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d17e5e753e64c4faf59dacb5ec6b4ab" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 43521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:44.282151 4653 consensus_queue.cc:1035] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "306af20b690a46bf83a8b269cb11cb81" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:15:44.285338 4663 mvcc.cc:205] Tried to move back new op lower bound from 7317286683773210624 to 7317286683688312832. Current Snapshot: MvccSnapshot[applied={T|T < 7317286683773210624}]
I20260811 11:15:44.286088 4664 mvcc.cc:205] Tried to move back new op lower bound from 7317286683773210624 to 7317286683688312832. Current Snapshot: MvccSnapshot[applied={T|T < 7317286683773210624 or (T in {7317286683773210624})}]
I20260811 11:15:44.288630 3191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:44.288753 3191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:15:44.290313 3191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d9f13a3038847b7a4c0a65ecae990c4: Sending DeleteTablet for 3 replicas of tablet 2b219f5d5f1a4d70b8d96eeabecd7db9
I20260811 11:15:44.290727 4541 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b219f5d5f1a4d70b8d96eeabecd7db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:44 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:56552
I20260811 11:15:44.290822 4384 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b219f5d5f1a4d70b8d96eeabecd7db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:44 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:53882
I20260811 11:15:44.291090 4673 tablet_replica.cc:330] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: stopping tablet replica
I20260811 11:15:44.291108 4228 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b219f5d5f1a4d70b8d96eeabecd7db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:15:44 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:60392
I20260811 11:15:44.291113 4674 tablet_replica.cc:330] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: stopping tablet replica
I20260811 11:15:44.291220 4673 raft_consensus.cc:2229] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:44.291332 4673 raft_consensus.cc:2258] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:44.291448 4674 raft_consensus.cc:2229] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:15:44.291955 4674 raft_consensus.cc:2258] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:44.292227 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 3157
I20260811 11:15:44.292243 4675 tablet_replica.cc:330] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: stopping tablet replica
I20260811 11:15:44.292443 4673 ts_tablet_manager.cc:1905] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:44.294569 4675 raft_consensus.cc:2229] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:15:44.294736 4675 raft_consensus.cc:2258] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:15:44.294881 4673 ts_tablet_manager.cc:1918] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:15:44.295011 4673 log.cc:1196] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/2b219f5d5f1a4d70b8d96eeabecd7db9
I20260811 11:15:44.295389 4673 ts_tablet_manager.cc:1939] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 4d17e5e753e64c4faf59dacb5ec6b4ab: Deleting consensus metadata
I20260811 11:15:44.295440 4675 ts_tablet_manager.cc:1905] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:44.297508 4675 ts_tablet_manager.cc:1918] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:15:44.297581 4675 log.cc:1196] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/2b219f5d5f1a4d70b8d96eeabecd7db9
I20260811 11:15:44.297873 4675 ts_tablet_manager.cc:1939] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 9d3925f34ae44afb9de4bcb773260034: Deleting consensus metadata
I20260811 11:15:44.305519 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.291968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.28.25.60:43933)
0811 11:15:44.292012 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.292014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.295715 (+ 3701us) server_negotiation.cc:407] Connection header received
0811 11:15:44.296286 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.296287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.296305 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.296335 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.297231 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.298079 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.299307 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.299630 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.299676 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.302049 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.302050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.302051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.302060 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.304047 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.304502 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.304503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.304504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.304574 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.304807 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.304808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.304808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.304986 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.305101 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.305259 (+ 158us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.305369 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260811 11:15:44.305862 4172 connection.cc:535] server connection from 127.0.0.1:60392 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 11:15:44.306216 4491 connection.cc:535] server connection from 127.0.0.1:56552 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 11:15:44.307204 4674 ts_tablet_manager.cc:1905] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:15:44.309350 4674 ts_tablet_manager.cc:1918] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:15:44.309440 4674 log.cc:1196] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/2b219f5d5f1a4d70b8d96eeabecd7db9
I20260811 11:15:44.309739 4674 ts_tablet_manager.cc:1939] T 2b219f5d5f1a4d70b8d96eeabecd7db9 P 306af20b690a46bf83a8b269cb11cb81: Deleting consensus metadata
W20260811 11:15:44.310185 4327 connection.cc:441] server connection from 127.0.0.1:53882 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53882 (request call id 1) could send its response
I20260811 11:15:44.316061 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.307869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.28.25.60:43933)
0811 11:15:44.307907 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.307909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.308041 (+ 132us) server_negotiation.cc:407] Connection header received
0811 11:15:44.308145 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.308146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.308158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.308188 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.309013 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.309707 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.310602 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.310928 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.311050 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.312712 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.312713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.312713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.312721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.314515 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.314948 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.314949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.314950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.315048 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.315288 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.315289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.315290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.315432 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.315525 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.315869 (+ 344us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.315952 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 11:15:44.318051 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:44.309411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.28.25.61:41245)
0811 11:15:44.309455 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 11:15:44.309456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:15:44.310703 (+ 1247us) server_negotiation.cc:407] Connection header received
0811 11:15:44.310810 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:15:44.310811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:15:44.310828 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:15:44.310860 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:15:44.311779 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.312397 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.313418 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:15:44.313753 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:15:44.313807 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:15:44.315253 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:15:44.315254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:15:44.315254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:15:44.315262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:15:44.316871 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.317226 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.317227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.317228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.317289 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:15:44.317506 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:15:44.317507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:15:44.317507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:15:44.317629 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:15:44.317738 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 11:15:44.317845 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 11:15:44.317944 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 11:15:45.287515 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:45.287180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:50511)
0811 11:15:45.287345 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:45.287450 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
W20260811 11:15:45.287806 4464 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:36971 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
I20260811 11:15:45.287884 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:45.287553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:60421)
0811 11:15:45.287712 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:45.287842 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
W20260811 11:15:45.288113 4302 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:36971 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
I20260811 11:15:45.289112 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:45.288850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:35737)
0811 11:15:45.288986 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:45.289059 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
W20260811 11:15:45.289296 4621 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:36971 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
I20260811 11:15:45.790997 4687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5d9f13a3038847b7a4c0a65ecae990c4)
I20260811 11:15:45.791186 4687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:15:45.791504 4687 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971)
I20260811 11:15:45.791958 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:45.791737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:56480)
0811 11:15:45.791850 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:45.791918 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
W20260811 11:15:45.792107 3863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
I20260811 11:15:45.792510 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:45.792393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:56492)
0811 11:15:45.792453 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:45.792481 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 11:15:45.792613 3863 leader_election.cc:334] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971): Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
I20260811 11:15:45.792654 3863 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6dbb9aa428404b838f3036368e7f4b77; no voters: 5d9f13a3038847b7a4c0a65ecae990c4
I20260811 11:15:45.792754 4687 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:15:46.288430 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:46.288142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:60043)
0811 11:15:46.288332 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:46.288378 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260811 11:15:46.288686 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:46.288461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:41829)
0811 11:15:46.288591 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:46.288627 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260811 11:15:46.289836 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:46.289596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:39489)
0811 11:15:46.289700 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:46.289737 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
I20260811 11:15:47.289270 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:47.288996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:59941)
0811 11:15:47.289151 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:47.289219 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260811 11:15:47.289430 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:47.289147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:54623)
0811 11:15:47.289309 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:47.289377 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260811 11:15:47.290555 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:47.290378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:40005)
0811 11:15:47.290483 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:47.290512 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260811 11:15:47.351594 4697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5d9f13a3038847b7a4c0a65ecae990c4)
I20260811 11:15:47.351712 4697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:15:47.351965 4697 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971)
I20260811 11:15:47.352470 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:47.352200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:56682)
0811 11:15:47.352385 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:47.352426 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260811 11:15:47.353022 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:47.352908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:56686)
0811 11:15:47.352964 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:47.352999 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:15:47.353106 3863 leader_election.cc:334] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971): Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
I20260811 11:15:47.353152 3863 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6dbb9aa428404b838f3036368e7f4b77; no voters: 5d9f13a3038847b7a4c0a65ecae990c4
I20260811 11:15:47.353232 4697 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:15:48.290087 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:48.289828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:60527)
0811 11:15:48.289961 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:48.290017 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260811 11:15:48.290331 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:48.289776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:52107)
0811 11:15:48.290183 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:48.290260 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260811 11:15:48.291266 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:48.291061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:50669)
0811 11:15:48.291189 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:48.291225 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260811 11:15:49.291047 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:49.290741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:60657)
0811 11:15:49.290894 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:49.290949 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
I20260811 11:15:49.291273 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:49.291056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:39643)
0811 11:15:49.291177 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:49.291209 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":52,"threads_started":1}
I20260811 11:15:49.291909 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:49.291685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:37347)
0811 11:15:49.291838 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:49.291871 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":46,"threads_started":1}
I20260811 11:15:49.683542 4707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5d9f13a3038847b7a4c0a65ecae990c4)
I20260811 11:15:49.683660 4707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:15:49.683889 4707 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971)
I20260811 11:15:49.684345 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:49.684132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:34888)
0811 11:15:49.684266 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:49.684304 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260811 11:15:49.684844 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:49.684747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:34900)
0811 11:15:49.684798 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:49.684824 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:15:49.684939 3863 leader_election.cc:334] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971): Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
I20260811 11:15:49.684984 3863 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6dbb9aa428404b838f3036368e7f4b77; no voters: 5d9f13a3038847b7a4c0a65ecae990c4
I20260811 11:15:49.685050 4707 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:15:50.291993 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:50.291729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:56579)
0811 11:15:50.291927 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:50.291955 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":127,"threads_started":1}
I20260811 11:15:50.291991 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:50.291729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:59665)
0811 11:15:50.291886 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:50.291933 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260811 11:15:50.292588 4712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:50.292321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:39429)
0811 11:15:50.292505 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:50.292547 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":51,"threads_started":1}
I20260811 11:15:51.292680 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:51.292434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:40961)
0811 11:15:51.292568 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:51.292611 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260811 11:15:51.292729 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:51.292439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:47857)
0811 11:15:51.292631 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:51.292672 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":98,"threads_started":1}
I20260811 11:15:51.293221 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:51.293030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:52729)
0811 11:15:51.293149 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:51.293184 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260811 11:15:52.293510 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:52.293149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:49143)
0811 11:15:52.293390 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:52.293449 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":72,"threads_started":1}
I20260811 11:15:52.293594 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:52.293213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:37869)
0811 11:15:52.293488 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:52.293534 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":67,"threads_started":1}
I20260811 11:15:52.293877 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:52.293620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:39307)
0811 11:15:52.293791 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:52.293832 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":57,"threads_started":1}
I20260811 11:15:53.156236 4719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5d9f13a3038847b7a4c0a65ecae990c4)
I20260811 11:15:53.156374 4719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:15:53.156544 4719 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971)
I20260811 11:15:53.156931 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:53.156709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:35150)
0811 11:15:53.156831 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:53.156881 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
W20260811 11:15:53.157074 3863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260811 11:15:53.157558 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:53.157456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:35152)
0811 11:15:53.157507 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:53.157538 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 11:15:53.157692 3863 leader_election.cc:334] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971): Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
I20260811 11:15:53.157763 3863 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6dbb9aa428404b838f3036368e7f4b77; no voters: 5d9f13a3038847b7a4c0a65ecae990c4
I20260811 11:15:53.157868 4719 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:15:53.294524 4722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:53.294146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:54455)
0811 11:15:53.294433 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:53.294473 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":79,"threads_started":1}
I20260811 11:15:53.294603 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:53.294159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:45601)
0811 11:15:53.294332 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:53.294386 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
I20260811 11:15:53.294741 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:53.294493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:52139)
0811 11:15:53.294638 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:53.294674 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260811 11:15:54.295646 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:54.295313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:55191)
0811 11:15:54.295528 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:54.295582 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
I20260811 11:15:54.295678 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:54.295079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:48419)
0811 11:15:54.295578 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:54.295622 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":84,"threads_started":1}
I20260811 11:15:54.295768 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:54.295559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:60553)
0811 11:15:54.295686 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:54.295724 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260811 11:15:55.296460 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:55.296180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:60639)
0811 11:15:55.296347 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:55.296394 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260811 11:15:55.296481 4729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:55.296180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:49823)
0811 11:15:55.296347 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:55.296441 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260811 11:15:55.296586 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:55.296352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:39389)
0811 11:15:55.296464 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:55.296508 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260811 11:15:56.297276 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:56.297064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:40087)
0811 11:15:56.297204 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:56.297238 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260811 11:15:56.297276 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:56.297021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:45889)
0811 11:15:56.297154 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:56.297208 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260811 11:15:56.297474 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:56.297166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:35805)
0811 11:15:56.297326 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:56.297415 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260811 11:15:56.519580 4735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5d9f13a3038847b7a4c0a65ecae990c4)
I20260811 11:15:56.519737 4735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:15:56.520008 4735 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971)
I20260811 11:15:56.520455 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:56.520221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:35326)
0811 11:15:56.520372 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:56.520414 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260811 11:15:56.521183 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:56.520970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:35336)
0811 11:15:56.521074 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:56.521148 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260811 11:15:56.521333 3863 leader_election.cc:334] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971): Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
I20260811 11:15:56.521386 3863 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6dbb9aa428404b838f3036368e7f4b77; no voters: 5d9f13a3038847b7a4c0a65ecae990c4
I20260811 11:15:56.521456 4735 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:15:57.298210 4739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:57.297882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:34183)
0811 11:15:57.298133 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:57.298164 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":71,"threads_started":1}
I20260811 11:15:57.298210 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:57.297992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:34673)
0811 11:15:57.298107 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:57.298153 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260811 11:15:57.298297 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:57.297882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:36717)
0811 11:15:57.298181 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:57.298240 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":71,"threads_started":1}
I20260811 11:15:58.299116 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:58.298801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:51037)
0811 11:15:58.299036 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:58.299067 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":76,"threads_started":1}
I20260811 11:15:58.299115 4741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:58.298801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:34107)
0811 11:15:58.299008 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:58.299057 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260811 11:15:58.299220 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:58.298944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:40457)
0811 11:15:58.299114 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:58.299158 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260811 11:15:59.300110 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:59.299875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:46777)
0811 11:15:59.300008 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:59.300053 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260811 11:15:59.300110 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:59.299732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:53455)
0811 11:15:59.300008 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:59.300053 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":70,"threads_started":1}
I20260811 11:15:59.300196 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:15:59.299957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:58949)
0811 11:15:59.300096 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0811 11:15:59.300137 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260811 11:16:00.300921 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:00.300631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:43663)
0811 11:16:00.300814 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:00.300872 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260811 11:16:00.300921 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:00.300632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:58823)
0811 11:16:00.300825 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:00.300863 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260811 11:16:00.300958 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:00.300686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:53589)
0811 11:16:00.300855 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:00.300900 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":60,"threads_started":1}
I20260811 11:16:01.301682 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:01.301345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:58979)
0811 11:16:01.301572 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:01.301618 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":86,"threads_started":1}
I20260811 11:16:01.301807 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:01.301390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:51295)
0811 11:16:01.301714 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:01.301756 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":67,"threads_started":1}
I20260811 11:16:01.301820 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:01.301634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:34961)
0811 11:16:01.301749 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:01.301781 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260811 11:16:02.302867 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:02.302422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:44513)
0811 11:16:02.302738 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:02.302801 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":87,"threads_started":1}
I20260811 11:16:02.303033 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:02.302566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:60895)
0811 11:16:02.302915 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:02.302982 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":60,"threads_started":1}
I20260811 11:16:02.303112 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:02.302573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:56767)
0811 11:16:02.303021 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:02.303064 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":182,"threads_started":1}
I20260811 11:16:03.303687 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:03.303428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:38717)
0811 11:16:03.303575 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:03.303628 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260811 11:16:03.303822 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:03.303549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:51329)
0811 11:16:03.303729 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:03.303768 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":52,"threads_started":1}
I20260811 11:16:03.303920 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:03.303657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:48031)
0811 11:16:03.303822 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:03.303873 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":60,"threads_started":1}
I20260811 11:16:04.240302 4763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5d9f13a3038847b7a4c0a65ecae990c4)
I20260811 11:16:04.240471 4763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5d9f13a3038847b7a4c0a65ecae990c4" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36971 } } peers { permanent_uuid: "6dbb9aa428404b838f3036368e7f4b77" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41245 } attrs { promote: false } } peers { permanent_uuid: "e90bb810d2c441c286193ff7e79d1075" member_type: NON_VOTER last_known_addr { host: "127.28.25.60" port: 43933 } attrs { promote: true } }
I20260811 11:16:04.240694 4763 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971)
I20260811 11:16:04.241154 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:04.240864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:36714)
0811 11:16:04.241065 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:04.241111 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
W20260811 11:16:04.241276 3863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260811 11:16:04.241891 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:04.241758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.0.0.1:36726)
0811 11:16:04.241827 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:04.241866 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 11:16:04.242014 3863 leader_election.cc:334] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5d9f13a3038847b7a4c0a65ecae990c4 (127.28.25.62:36971): Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
I20260811 11:16:04.242098 3863 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6dbb9aa428404b838f3036368e7f4b77; no voters: 5d9f13a3038847b7a4c0a65ecae990c4
I20260811 11:16:04.242228 4763 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6dbb9aa428404b838f3036368e7f4b77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:16:04.304661 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:04.304344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:45643)
0811 11:16:04.304558 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:04.304612 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260811 11:16:04.304716 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:04.304352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:60663)
0811 11:16:04.304615 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:04.304660 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":95,"threads_started":1}
I20260811 11:16:04.304739 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:04.304267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:59387)
0811 11:16:04.304665 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:04.304699 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":73,"threads_started":1}
I20260811 11:16:05.305517 4770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:05.305183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:45777)
0811 11:16:05.305396 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:05.305454 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260811 11:16:05.305730 4771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:05.305251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:45115)
0811 11:16:05.305637 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:05.305682 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":64,"threads_started":1}
I20260811 11:16:05.305737 4772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:05.305406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:34963)
0811 11:16:05.305667 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:05.305698 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":92,"threads_started":1}
I20260811 11:16:06.306568 4776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:06.306216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:51589)
0811 11:16:06.306410 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:06.306461 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20260811 11:16:06.306597 4777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:06.306275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:49873)
0811 11:16:06.306498 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:06.306542 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":57,"threads_started":1}
I20260811 11:16:06.306816 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:06.306201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:51485)
0811 11:16:06.306712 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:06.306749 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":111,"threads_started":1}
I20260811 11:16:07.307723 4780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:07.307399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:47937)
0811 11:16:07.307605 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:07.307658 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":66,"threads_started":1}
I20260811 11:16:07.307792 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:07.307233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:44917)
0811 11:16:07.307689 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:07.307736 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260811 11:16:07.307864 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:07.307334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:41429)
0811 11:16:07.307724 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:07.307760 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":61,"threads_started":1}
I20260811 11:16:08.308864 4781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:08.308600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:52137)
0811 11:16:08.308742 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:08.308795 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260811 11:16:08.309115 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:08.308912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:56571)
0811 11:16:08.309025 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:08.309062 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260811 11:16:08.309244 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:08.308990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:55761)
0811 11:16:08.309153 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:08.309188 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260811 11:16:09.309793 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:09.309550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:43311)
0811 11:16:09.309697 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:09.309738 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260811 11:16:09.309793 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:09.309550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:53217)
0811 11:16:09.309697 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:09.309739 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260811 11:16:09.309899 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:09.309636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:46863)
0811 11:16:09.309789 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:09.309832 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260811 11:16:10.310521 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:10.310251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:48143)
0811 11:16:10.310400 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:10.310447 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260811 11:16:10.310575 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:10.310319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:33989)
0811 11:16:10.310486 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:10.310523 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260811 11:16:10.310736 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:10.310553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:55925)
0811 11:16:10.310656 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:10.310685 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
I20260811 11:16:11.311299 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:11.310953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:32951)
0811 11:16:11.311147 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:11.311232 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260811 11:16:11.311313 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:11.311017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:46719)
0811 11:16:11.311201 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:11.311250 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":53,"threads_started":1}
I20260811 11:16:11.311477 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:11.311196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:34753)
0811 11:16:11.311352 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:11.311432 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":50,"threads_started":1}
I20260811 11:16:12.312170 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:12.311893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:54009)
0811 11:16:12.312056 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:12.312109 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260811 11:16:12.312211 4795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:12.311893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:49209)
0811 11:16:12.312129 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:12.312163 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":100,"threads_started":1}
I20260811 11:16:12.312506 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:12.311949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:42423)
0811 11:16:12.312365 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:12.312426 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":84,"threads_started":1}
I20260811 11:16:13.313033 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:13.312725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:42749)
0811 11:16:13.312924 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:13.312973 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":61,"threads_started":1}
I20260811 11:16:13.313119 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:13.312716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:35325)
0811 11:16:13.313013 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:13.313060 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":70,"threads_started":1}
I20260811 11:16:13.313393 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:13.313080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:53817)
0811 11:16:13.313288 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:13.313336 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":66,"threads_started":1}
I20260811 11:16:14.313890 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:14.313606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.3:33891)
0811 11:16:14.313776 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:14.313822 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260811 11:16:14.314087 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:14.313607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.1:43689)
0811 11:16:14.313872 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:14.313970 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":87,"threads_started":1}
I20260811 11:16:14.314270 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:14.314032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:36971 (local address 127.28.25.2:34359)
0811 11:16:14.314181 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:14.314217 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.28.25.62:36971,127.28.25.61:41245,127.28.25.60:43933, num_attempts: 242) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.28.25.62:36971,127.28.25.61:41245,127.28.25.60:43933, num_attempts: 1)
I20260811 11:16:14.317625 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 4039
I20260811 11:16:14.328954 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 4310
I20260811 11:16:14.336319 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 4470
I20260811 11:16:14.345474 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 3725
I20260811 11:16:14.353636 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 3935
2026-08-11T11:16:14Z chronyd exiting
I20260811 11:16:14.373519 28772 test_util.cc:175] -----------------------------------------------
I20260811 11:16:14.373588 28772 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786446911996770-28772-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33307 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:16:14 dist-test-slave-2w3w krb5kdc[4807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:16:14 dist-test-slave-2w3w krb5kdc[4807](info): set up 2 sockets
Aug 11 11:16:14 dist-test-slave-2w3w krb5kdc[4807](info): commencing operation
krb5kdc: starting...
W20260811 11:16:15.488392 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:16:15 dist-test-slave-2w3w krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:16:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.62.keytab.
Entry for principal oryx/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.62.keytab.
I20260811 11:16:15.631844 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:42787
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.62.keytab
--principal=oryx/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:15.727054 4821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:15.727378 4821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:15.727428 4821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:15.728890 4821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:16:15.728986 4821 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 11:16:15.729009 4821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:15.729038 4821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:15.729074 4821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:16:15.729106 4821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:16:15.731086 4821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:42787
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.62.keytab
--principal=oryx/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:15.731421 4821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:15.731693 4821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:15.738114 4827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:15.738121 4826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:15.738483 4829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:15.738719 4821 server_base.cc:1034] running on GCE node
I20260811 11:16:15.739105 4821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:15.739502 4821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:15.740687 4821 hybrid_clock.cc:648] HybridClock initialized: now 1786446975740668 us; error 35 us; skew 500 ppm
Aug 11 11:16:15 dist-test-slave-2w3w krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446975, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:15.743803 4821 init.cc:377] Logged in from keytab as oryx/127.28.25.62@KRBTEST.COM (short username oryx)
I20260811 11:16:15.744366 4821 webserver.cc:460] Webserver started at http://127.28.25.62:44389/ using document root <none> and password file <none>
I20260811 11:16:15.744618 4821 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:15.744668 4821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:15.744858 4821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:15.746079 4821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "631a0d95ba624836a69709ab8f794a79"
format_stamp: "Formatted at 2026-08-11 11:16:15 on dist-test-slave-2w3w"
server_key: "297a4fa6a6ef14415689be8a301c81e5"
server_key_iv: "cb1d508310c5325f04ec71dd85d279e3"
server_key_version: "encryptionkey@0"
I20260811 11:16:15.746486 4821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "631a0d95ba624836a69709ab8f794a79"
format_stamp: "Formatted at 2026-08-11 11:16:15 on dist-test-slave-2w3w"
server_key: "297a4fa6a6ef14415689be8a301c81e5"
server_key_iv: "cb1d508310c5325f04ec71dd85d279e3"
server_key_version: "encryptionkey@0"
I20260811 11:16:15.748150 4821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:16:15.749115 4836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:15.749485 4821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:16:15.749588 4821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "631a0d95ba624836a69709ab8f794a79"
format_stamp: "Formatted at 2026-08-11 11:16:15 on dist-test-slave-2w3w"
server_key: "297a4fa6a6ef14415689be8a301c81e5"
server_key_iv: "cb1d508310c5325f04ec71dd85d279e3"
server_key_version: "encryptionkey@0"
I20260811 11:16:15.749707 4821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:15.770375 4821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:15.772969 4821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:15.773138 4821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:15.778800 4821 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:42787
I20260811 11:16:15.778846 4907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:42787 every 8 connection(s)
I20260811 11:16:15.779281 4821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:16:15.780021 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:15.782472 4908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79: Bootstrap starting.
I20260811 11:16:15.783196 4908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:15.783481 4908 log.cc:824] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:15.784277 4908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79: No bootstrap required, opened a new log
I20260811 11:16:15.786576 4908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a0d95ba624836a69709ab8f794a79" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42787 } }
I20260811 11:16:15.786751 4908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:15.786775 4908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631a0d95ba624836a69709ab8f794a79, State: Initialized, Role: FOLLOWER
I20260811 11:16:15.786952 4908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a0d95ba624836a69709ab8f794a79" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42787 } }
I20260811 11:16:15.787081 4908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:16:15.787109 4908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:16:15.787148 4908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:15.787778 4908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a0d95ba624836a69709ab8f794a79" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42787 } }
I20260811 11:16:15.787886 4908 leader_election.cc:302] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 631a0d95ba624836a69709ab8f794a79; no voters:
I20260811 11:16:15.788048 4908 leader_election.cc:288] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:16:15.788123 4911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:15.788331 4911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [term 1 LEADER]: Becoming Leader. State: Replica: 631a0d95ba624836a69709ab8f794a79, State: Running, Role: LEADER
I20260811 11:16:15.788394 4908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:16:15.788458 4911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a0d95ba624836a69709ab8f794a79" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42787 } }
I20260811 11:16:15.788932 4914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 631a0d95ba624836a69709ab8f794a79. Latest consensus state: current_term: 1 leader_uuid: "631a0d95ba624836a69709ab8f794a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a0d95ba624836a69709ab8f794a79" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42787 } } }
I20260811 11:16:15.789032 4914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:15.789191 4913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "631a0d95ba624836a69709ab8f794a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a0d95ba624836a69709ab8f794a79" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42787 } } }
I20260811 11:16:15.789322 4913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:15.789386 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 4821
I20260811 11:16:15.789496 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:16:15.789714 28772 external_mini_cluster.cc:1428] Setting key 0350658c8cc53e6b7ca394a01a36abcf
I20260811 11:16:15.789812 4923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:16:15.791049 4923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:16:15.792840 4923 catalog_manager.cc:1347] Generated new cluster ID: c2a7e3a145d944ca8b88ea948468a036
I20260811 11:16:15.792902 4923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 11:16:15 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:16:15.802677 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:15.790248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.28.25.62:42787)
0811 11:16:15.790495 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:15.790506 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:15.790547 (+ 41us) server_negotiation.cc:407] Connection header received
0811 11:16:15.790667 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:15.790673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:15.790720 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:15.790871 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:15.791635 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:15.792917 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:15.794029 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:15.794757 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:15.794899 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:15.797933 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:15.797939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:15.797939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:15.797955 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:15.800138 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:15.800667 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:15.800671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:15.800672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:15.801068 (+ 396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:15.801415 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:15.801416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:15.801416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:15.801793 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:15.801969 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:15.802121 (+ 152us) server_negotiation.cc:299] Negotiation successful
0811 11:16:15.802414 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":110,"threads_started":1}
I20260811 11:16:15.814602 4923 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:16:15.815521 4923 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:16:15.824961 4923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79: Generated new TSK 0
I20260811 11:16:15.825222 4923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab.
Entry for principal oryx/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab.
I20260811 11:16:15.863900 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42787
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab
--principal=oryx/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:15.954391 4939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:15.954618 4939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:15.954648 4939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:15.955984 4939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:15.956048 4939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:15.956069 4939 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:16:15.956089 4939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:16:15.957810 4939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab
--principal=oryx/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.25.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:15.958109 4939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:15.958395 4939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:15.964258 4946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:15.964236 4945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:15.964367 4939 server_base.cc:1034] running on GCE node
W20260811 11:16:15.964236 4948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:15.964916 4939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:15.965226 4939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:15.966394 4939 hybrid_clock.cc:648] HybridClock initialized: now 1786446975966361 us; error 54 us; skew 500 ppm
Aug 11 11:16:15 dist-test-slave-2w3w krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446975, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:15.969434 4939 init.cc:377] Logged in from keytab as oryx/127.28.25.1@KRBTEST.COM (short username oryx)
I20260811 11:16:15.969959 4939 webserver.cc:460] Webserver started at http://127.28.25.1:39181/ using document root <none> and password file <none>
I20260811 11:16:15.970183 4939 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:15.970227 4939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:15.970331 4939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:15.971509 4939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "980e7592509d4f6ba4608a6f60076edc"
format_stamp: "Formatted at 2026-08-11 11:16:15 on dist-test-slave-2w3w"
server_key: "7c4d9e966da7053679f4c5b173be73d2"
server_key_iv: "b45097138a62076e5e2bfd45028cc1d9"
server_key_version: "encryptionkey@0"
I20260811 11:16:15.971846 4939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "980e7592509d4f6ba4608a6f60076edc"
format_stamp: "Formatted at 2026-08-11 11:16:15 on dist-test-slave-2w3w"
server_key: "7c4d9e966da7053679f4c5b173be73d2"
server_key_iv: "b45097138a62076e5e2bfd45028cc1d9"
server_key_version: "encryptionkey@0"
I20260811 11:16:15.973366 4939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:16:15.974400 4955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:15.974606 4939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:15.974738 4939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "980e7592509d4f6ba4608a6f60076edc"
format_stamp: "Formatted at 2026-08-11 11:16:15 on dist-test-slave-2w3w"
server_key: "7c4d9e966da7053679f4c5b173be73d2"
server_key_iv: "b45097138a62076e5e2bfd45028cc1d9"
server_key_version: "encryptionkey@0"
I20260811 11:16:15.974833 4939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:15.999027 4939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:16.001724 4939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:16.001929 4939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:16.002631 4939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:16.002688 4939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:16.002730 4939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:16.002750 4939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:16.011538 4939 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:38073
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446975, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:16:16.011721 5090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:38073 every 8 connection(s)
I20260811 11:16:16.012140 4939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:16:16.016095 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.003327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:49707 (local address 127.28.25.62:42787)
0811 11:16:16.003411 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.003413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.003663 (+ 250us) server_negotiation.cc:407] Connection header received
0811 11:16:16.004293 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.004294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.004309 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.004363 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.005466 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.006332 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.008247 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.008508 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.008626 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.012346 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:16.012349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:16.012349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:16.012364 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:16.014114 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.014663 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.014664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.014664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.014743 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.015249 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.015250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.015250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.015399 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:16.015472 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.015900 (+ 428us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.016021 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:16:16.016260 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.003308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42787 (local address 127.28.25.1:49707)
0811 11:16:16.003518 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.003562 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.004155 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.004500 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.004505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.004966 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:16.005322 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.005328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.006482 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.006483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.008027 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.008031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.008737 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.008739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.008934 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.009642 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:16.009652 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:16.012132 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:16.014234 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.014243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.014245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.014566 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.014918 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.014920 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.014920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.015143 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.015527 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:16.015536 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:16.015696 (+ 160us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.015805 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.015969 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":127,"threads_started":1}
I20260811 11:16:16.016903 5092 heartbeater.cc:347] Connected to a master server at 127.28.25.62:42787
I20260811 11:16:16.017027 5092 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:16.017230 5092 heartbeater.cc:511] Master 127.28.25.62:42787 requested a full tablet report, sending...
I20260811 11:16:16.017853 4857 ts_manager.cc:194] Registered new tserver with Master: 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073)
I20260811 11:16:16.018790 4857 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:49707
I20260811 11:16:16.020897 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 4939
I20260811 11:16:16.021066 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:16:16.021344 28772 external_mini_cluster.cc:1428] Setting key 5667b4bc478d2f1c53deef9b599459f8
I20260811 11:16:16.025588 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.018295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42787 (local address 127.28.25.1:56607)
0811 11:16:16.018490 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.018502 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.018572 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.018716 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.018716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.019005 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:16.019090 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.019093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.020073 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.020075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.020652 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.020654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.021357 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.021358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.021426 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.022023 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:16.022028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:16.022497 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:16.024414 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.024416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.024416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.024615 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.024853 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.024854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.024854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.024926 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.025198 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:16.025199 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:16.025256 (+ 57us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.025345 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.025480 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260811 11:16:16.025585 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.018386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:56607 (local address 127.28.25.62:42787)
0811 11:16:16.018439 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.018442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.018597 (+ 155us) server_negotiation.cc:407] Connection header received
0811 11:16:16.018622 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.018623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.018633 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.018689 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.019160 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.019900 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.020755 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.021197 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.021288 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.022590 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:16.022591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:16.022591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:16.022598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:16.024291 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.024699 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.024700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.024701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.024770 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.024995 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.024996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.024996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.025123 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:16.025208 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.025384 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.025480 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
WARNING: no policy specified for oryx/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab.
Entry for principal oryx/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab.
I20260811 11:16:16.055209 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42787
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab
--principal=oryx/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:16.143887 5102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:16.144152 5102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:16.144183 5102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:16.145546 5102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:16.145598 5102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:16.145622 5102 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:16:16.145643 5102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:16:16.147470 5102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab
--principal=oryx/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.25.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:16.147745 5102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:16.147967 5102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:16.154026 5108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:16.154026 5110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:16.154062 5107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:16.154630 5102 server_base.cc:1034] running on GCE node
I20260811 11:16:16.154903 5102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:16.155332 5102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:16.156509 5102 hybrid_clock.cc:648] HybridClock initialized: now 1786446976156487 us; error 49 us; skew 500 ppm
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446976, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:16.159593 5102 init.cc:377] Logged in from keytab as oryx/127.28.25.2@KRBTEST.COM (short username oryx)
I20260811 11:16:16.160238 5102 webserver.cc:460] Webserver started at http://127.28.25.2:38681/ using document root <none> and password file <none>
I20260811 11:16:16.160526 5102 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:16.160581 5102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:16.160724 5102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:16.161979 5102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2"
format_stamp: "Formatted at 2026-08-11 11:16:16 on dist-test-slave-2w3w"
server_key: "5d3dbe002176285f7563c3724f3517c7"
server_key_iv: "7327979b80b6d4972a0fac9473de9c19"
server_key_version: "encryptionkey@0"
I20260811 11:16:16.162417 5102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2"
format_stamp: "Formatted at 2026-08-11 11:16:16 on dist-test-slave-2w3w"
server_key: "5d3dbe002176285f7563c3724f3517c7"
server_key_iv: "7327979b80b6d4972a0fac9473de9c19"
server_key_version: "encryptionkey@0"
I20260811 11:16:16.164588 5102 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 11:16:16.165624 5117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:16.165933 5102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:16.166050 5102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2"
format_stamp: "Formatted at 2026-08-11 11:16:16 on dist-test-slave-2w3w"
server_key: "5d3dbe002176285f7563c3724f3517c7"
server_key_iv: "7327979b80b6d4972a0fac9473de9c19"
server_key_version: "encryptionkey@0"
I20260811 11:16:16.166122 5102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:16.183506 5102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:16.186162 5102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:16.186311 5102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:16.186815 5102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:16.186854 5102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:16.186878 5102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:16.186892 5102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446976, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:16:16.195057 5102 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:44577
I20260811 11:16:16.195148 5247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:44577 every 8 connection(s)
I20260811 11:16:16.195495 5102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:16:16.199009 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.187586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:35217 (local address 127.28.25.62:42787)
0811 11:16:16.187664 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.187667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.187895 (+ 228us) server_negotiation.cc:407] Connection header received
0811 11:16:16.188492 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.188494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.188509 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.188597 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.189541 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.190470 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.192327 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.192654 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.192731 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.195615 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:16.195616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:16.195617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:16.195626 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:16.197225 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.197739 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.197741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.197741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.197809 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.198196 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.198197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.198197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.198341 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:16.198425 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.198763 (+ 338us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.198867 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:16:16.199239 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.187530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42787 (local address 127.28.25.2:35217)
0811 11:16:16.187742 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.187788 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.188345 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.188709 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.188715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.189182 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:16.189413 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.189419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.190574 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.190575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.192212 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.192217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.192747 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.192748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.192919 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.193628 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:16.193639 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:16.195473 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:16.197339 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.197350 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.197352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.197632 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.197963 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.197964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.197965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.198115 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.198417 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:16.198427 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:16.198610 (+ 183us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.198736 (+ 126us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.198939 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":129,"threads_started":1}
I20260811 11:16:16.199932 5248 heartbeater.cc:347] Connected to a master server at 127.28.25.62:42787
I20260811 11:16:16.200052 5248 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:16.200275 5248 heartbeater.cc:511] Master 127.28.25.62:42787 requested a full tablet report, sending...
I20260811 11:16:16.200824 4858 ts_manager.cc:194] Registered new tserver with Master: 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577)
I20260811 11:16:16.201274 4858 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:35217
I20260811 11:16:16.202099 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 5102
I20260811 11:16:16.202246 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:16:16.202502 28772 external_mini_cluster.cc:1428] Setting key 7717942a0b5c02755f49e958651f3ded
I20260811 11:16:16.209018 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.201671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:36249 (local address 127.28.25.62:42787)
0811 11:16:16.201720 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.201722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.201891 (+ 169us) server_negotiation.cc:407] Connection header received
0811 11:16:16.201942 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.201943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.201956 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.202015 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.202458 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.203261 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.204097 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.204421 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.204499 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.205781 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:16.205782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:16.205783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:16.205790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:16.207500 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.208030 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.208031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.208032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.208099 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.208368 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.208369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.208369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.208547 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:16.208609 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.208820 (+ 211us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.208904 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:16:16.209074 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.201588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42787 (local address 127.28.25.2:36249)
0811 11:16:16.201791 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.201803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.201874 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.202028 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.202029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.202285 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:16.202349 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.202351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.203388 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.203390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.203993 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.203996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.204508 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.204509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.204569 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.205139 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:16.205143 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:16.205668 (+ 525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:16.207610 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.207611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.207612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.207888 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.208189 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.208190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.208191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.208287 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.208627 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:16.208629 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:16.208702 (+ 73us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.208788 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.208938 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for oryx/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab.
Entry for principal oryx/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab.
I20260811 11:16:16.240103 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42787
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab
--principal=oryx/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:16.331579 5257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:16.331866 5257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:16.331915 5257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:16.333283 5257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:16.333377 5257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:16.333415 5257 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:16:16.333499 5257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:16:16.335476 5257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab
--principal=oryx/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.25.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:16.335807 5257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:16.336088 5257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:16.342008 5263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:16.342008 5265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:16.342054 5257 server_base.cc:1034] running on GCE node
W20260811 11:16:16.342054 5262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:16.342561 5257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:16.342880 5257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:16.344089 5257 hybrid_clock.cc:648] HybridClock initialized: now 1786446976344063 us; error 44 us; skew 500 ppm
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446976, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:16.347266 5257 init.cc:377] Logged in from keytab as oryx/127.28.25.3@KRBTEST.COM (short username oryx)
I20260811 11:16:16.347815 5257 webserver.cc:460] Webserver started at http://127.28.25.3:38811/ using document root <none> and password file <none>
I20260811 11:16:16.348066 5257 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:16.348116 5257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:16.348295 5257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:16.349525 5257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "718226ccbc2742f3a8429238ae014ae8"
format_stamp: "Formatted at 2026-08-11 11:16:16 on dist-test-slave-2w3w"
server_key: "f8a8f9cb2ff963a14a5729763b10aecc"
server_key_iv: "2370d4747c6bddcd61f897d6d410f0c4"
server_key_version: "encryptionkey@0"
I20260811 11:16:16.349959 5257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "718226ccbc2742f3a8429238ae014ae8"
format_stamp: "Formatted at 2026-08-11 11:16:16 on dist-test-slave-2w3w"
server_key: "f8a8f9cb2ff963a14a5729763b10aecc"
server_key_iv: "2370d4747c6bddcd61f897d6d410f0c4"
server_key_version: "encryptionkey@0"
I20260811 11:16:16.351713 5257 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260811 11:16:16.352810 5272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:16.353044 5257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:16.353169 5257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "718226ccbc2742f3a8429238ae014ae8"
format_stamp: "Formatted at 2026-08-11 11:16:16 on dist-test-slave-2w3w"
server_key: "f8a8f9cb2ff963a14a5729763b10aecc"
server_key_iv: "2370d4747c6bddcd61f897d6d410f0c4"
server_key_version: "encryptionkey@0"
I20260811 11:16:16.353267 5257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:16.369850 5257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:16.372505 5257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:16.372712 5257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:16.373368 5257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:16.373461 5257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:16.373504 5257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:16.373551 5257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446976, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:16:16.381440 5257 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:37919
I20260811 11:16:16.381510 5401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:37919 every 8 connection(s)
I20260811 11:16:16.381865 5257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:16:16.385068 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.373912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:57075 (local address 127.28.25.62:42787)
0811 11:16:16.373992 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.373994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.374220 (+ 226us) server_negotiation.cc:407] Connection header received
0811 11:16:16.374763 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.374764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.374782 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.374866 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.375713 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.376546 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.378347 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.378697 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.378782 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.381908 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:16.381910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:16.381910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:16.381919 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:16.383530 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.384028 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.384029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.384030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.384089 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.384341 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.384342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.384342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.384487 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:16.384544 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.384912 (+ 368us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.384998 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:16:16.385383 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.373893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42787 (local address 127.28.25.3:57075)
0811 11:16:16.374072 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.374118 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.374640 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.375016 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.375021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.375369 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:16.375602 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.375607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.376641 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.376642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.378213 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.378217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.378833 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.378834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.379067 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.379687 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:16.379699 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:16.381775 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:16.383651 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.383660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.383662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.383941 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.384157 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.384157 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.384158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.384265 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.384587 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:16.384595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:16.384740 (+ 145us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.384884 (+ 144us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.385044 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260811 11:16:16.386089 5402 heartbeater.cc:347] Connected to a master server at 127.28.25.62:42787
I20260811 11:16:16.386212 5402 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:16.386437 5402 heartbeater.cc:511] Master 127.28.25.62:42787 requested a full tablet report, sending...
I20260811 11:16:16.386915 4858 ts_manager.cc:194] Registered new tserver with Master: 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3:37919)
I20260811 11:16:16.387156 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 5257
I20260811 11:16:16.387264 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:16:16.387450 4858 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:57075
I20260811 11:16:16.387490 28772 external_mini_cluster.cc:1428] Setting key d282d3e105d3498b607d035c113a84e6
I20260811 11:16:16.388388 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:16:16.396082 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.387510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42787 (local address 127.28.25.3:49543)
0811 11:16:16.387746 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.387757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.388314 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.388630 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.388631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.388859 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:16.388946 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.388949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.389785 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.389787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.390678 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.390682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.391071 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.391072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.391147 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.391713 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:16.391718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:16.392090 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:16.394194 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.394196 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.394197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.394476 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.395322 (+ 846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.395322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.395323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.395389 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.395739 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:16.395740 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:16.395788 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.395842 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.396002 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":157,"threads_started":1}
I20260811 11:16:16.396351 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.387660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:49543 (local address 127.28.25.62:42787)
0811 11:16:16.387707 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.387710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.387831 (+ 121us) server_negotiation.cc:407] Connection header received
0811 11:16:16.388549 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.388550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.388563 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.388621 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.389047 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.389682 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.390769 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.390954 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.391067 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.392193 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:16.392194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:16.392195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:16.392202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:16.394084 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.394592 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.394594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.394594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.394665 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.395468 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.395469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.395470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.395656 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:16.395750 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.395903 (+ 153us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.396013 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 11:16:16.416426 4933 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:16.409753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.28.25.62:42787)
0811 11:16:16.409838 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.409840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.409853 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:16:16.409881 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.409882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.409893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.409963 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.410629 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.411510 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.412476 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.412679 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.412765 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.416286 (+ 3521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40702: BlockingRecv error: recv got EOF from 127.0.0.1:40702 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:16:16.425372 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.417328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.28.25.62:42787)
0811 11:16:16.417402 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.417403 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.417415 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:16:16.417498 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.417499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.417509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.417569 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.418173 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.418677 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.419526 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.419723 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.419786 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.422407 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:16.422409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:16.422409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:16.422418 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:16.423957 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.424396 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.424398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.424398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.424502 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.424786 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.424788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.424788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.424960 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:16.425048 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.425223 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.425299 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260811 11:16:16.427892 4858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:16:16.429538 4858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:16:16.437125 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.432678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.28.25.2:44577)
0811 11:16:16.432986 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.432990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.433005 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:16:16.433053 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.433057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.433074 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.433413 (+ 339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:16.433484 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.434264 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.435906 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.436601 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.436703 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.436808 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.436972 (+ 164us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.437015 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":88,"threads_started":1}
I20260811 11:16:16.437115 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.432318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:44577 (local address 127.0.0.1:39820)
0811 11:16:16.432491 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.432541 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.432615 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.433216 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.433219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.433222 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:16.433317 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.433319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.434383 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.434384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.435796 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.435799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.436702 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.436702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.436815 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.436834 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.436945 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.437002 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260811 11:16:16.437116 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.432318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:38073 (local address 127.0.0.1:54618)
0811 11:16:16.432516 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.432541 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.432615 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.433230 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.433231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.433280 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:16.433354 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.433356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.434765 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.434766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.435850 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.435852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.436752 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.436754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.436822 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.436851 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.436909 (+ 58us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.436969 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":5,"thread_start_us":150,"threads_started":1}
I20260811 11:16:16.437506 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.432335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:37919 (local address 127.0.0.1:59292)
0811 11:16:16.433028 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.433040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.433109 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.433360 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.433361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.433363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:16.433399 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.433400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.435094 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.435094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.436157 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.436161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.437244 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.437245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.437315 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.437320 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.437384 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.437429 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":2,"thread_start_us":143,"threads_started":1}
I20260811 11:16:16.437657 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.432810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.28.25.1:38073)
0811 11:16:16.433016 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.433020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.433030 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:16:16.433066 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.433067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.433080 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.433442 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:16.433516 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.434666 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.435941 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.436643 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.437155 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.437454 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.437482 (+ 28us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.437532 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":116,"threads_started":1}
I20260811 11:16:16.437844 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.432636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.28.25.3:37919)
0811 11:16:16.432889 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.432895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.433126 (+ 231us) server_negotiation.cc:407] Connection header received
0811 11:16:16.433175 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.433177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.433194 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.433272 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:16.433486 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.434652 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.436266 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.437064 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.437129 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.437206 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.437681 (+ 475us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.437741 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":138,"threads_started":1}
I20260811 11:16:16.438381 5017 tablet_service.cc:1467] Processing CreateTablet for tablet 92b5936cc2ee4345b7ad6682dcb8f396 (DEFAULT_TABLE table=test-table [id=92de1498f76b423cabe3f4f6ceb6f744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:16.438373 5174 tablet_service.cc:1467] Processing CreateTablet for tablet 92b5936cc2ee4345b7ad6682dcb8f396 (DEFAULT_TABLE table=test-table [id=92de1498f76b423cabe3f4f6ceb6f744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:16.438791 5174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b5936cc2ee4345b7ad6682dcb8f396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.438791 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b5936cc2ee4345b7ad6682dcb8f396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.441578 5314 tablet_service.cc:1467] Processing CreateTablet for tablet 92b5936cc2ee4345b7ad6682dcb8f396 (DEFAULT_TABLE table=test-table [id=92de1498f76b423cabe3f4f6ceb6f744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:16.442041 5314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b5936cc2ee4345b7ad6682dcb8f396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.443252 5437 tablet_bootstrap.cc:492] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: Bootstrap starting.
I20260811 11:16:16.443635 5438 tablet_bootstrap.cc:492] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Bootstrap starting.
I20260811 11:16:16.444175 5437 tablet_bootstrap.cc:654] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.444262 5438 tablet_bootstrap.cc:654] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.444664 5438 log.cc:824] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:16.444792 5437 log.cc:824] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:16.445559 5437 tablet_bootstrap.cc:492] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: No bootstrap required, opened a new log
I20260811 11:16:16.445659 5437 ts_tablet_manager.cc:1397] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:16:16.445740 5438 tablet_bootstrap.cc:492] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: No bootstrap required, opened a new log
I20260811 11:16:16.445835 5438 ts_tablet_manager.cc:1397] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:16:16.446677 5441 tablet_bootstrap.cc:492] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: Bootstrap starting.
I20260811 11:16:16.447542 5441 tablet_bootstrap.cc:654] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.447618 5437 raft_consensus.cc:348] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.447824 5437 raft_consensus.cc:374] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.447876 5437 raft_consensus.cc:729] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e7592509d4f6ba4608a6f60076edc, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.447949 5441 log.cc:824] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:16.448027 5437 consensus_queue.cc:260] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.448352 5092 heartbeater.cc:503] Master 127.28.25.62:42787 was elected leader, sending a full tablet report...
I20260811 11:16:16.448341 5437 ts_tablet_manager.cc:1428] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:16:16.448580 5438 raft_consensus.cc:348] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.448807 5438 raft_consensus.cc:374] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.448853 5438 raft_consensus.cc:729] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.448920 5441 tablet_bootstrap.cc:492] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: No bootstrap required, opened a new log
I20260811 11:16:16.448990 5441 ts_tablet_manager.cc:1397] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 11:16:16.448976 5438 consensus_queue.cc:260] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.449292 5438 ts_tablet_manager.cc:1428] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260811 11:16:16.449447 5248 heartbeater.cc:503] Master 127.28.25.62:42787 was elected leader, sending a full tablet report...
I20260811 11:16:16.451191 5441 raft_consensus.cc:348] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.451404 5441 raft_consensus.cc:374] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.451473 5441 raft_consensus.cc:729] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718226ccbc2742f3a8429238ae014ae8, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.451789 5441 consensus_queue.cc:260] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.452123 5441 ts_tablet_manager.cc:1428] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260811 11:16:16.452252 5402 heartbeater.cc:503] Master 127.28.25.62:42787 was elected leader, sending a full tablet report...
I20260811 11:16:16.459440 5444 raft_consensus.cc:482] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:16:16.459592 5444 raft_consensus.cc:504] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.460000 5444 leader_election.cc:288] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073), 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3:37919)
I20260811 11:16:16.463639 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.460250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:37919 (local address 127.28.25.2:52505)
0811 11:16:16.460535 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.460547 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.460630 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.460805 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.460806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.460807 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:16.460867 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.460868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.461830 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.461831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.462646 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.462649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.463326 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.463327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.463414 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.463417 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.463486 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.463534 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":78,"thread_start_us":154,"threads_started":1}
I20260811 11:16:16.463617 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.460391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:52505 (local address 127.28.25.3:37919)
0811 11:16:16.460438 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.460440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.460602 (+ 162us) server_negotiation.cc:407] Connection header received
0811 11:16:16.460717 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.460718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.460732 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.460789 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:16.461040 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.461710 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.462759 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.463218 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.463307 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.463322 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.463495 (+ 173us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.463540 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:16:16.463734 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.460250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:38073 (local address 127.28.25.2:53275)
0811 11:16:16.460319 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.460335 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.460406 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.460597 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.460598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.460600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:16.460703 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.460706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.461708 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.461710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.462650 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.462653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.463411 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.463412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.463520 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.463523 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.463600 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.463652 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1}
I20260811 11:16:16.463750 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.460354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:53275 (local address 127.28.25.1:38073)
0811 11:16:16.460410 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.460412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.460450 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:16:16.460502 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.460503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.460518 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.460576 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:16.460777 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.461598 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.462781 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.463298 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.463407 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.463420 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.463622 (+ 202us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.463671 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 11:16:16.464090 5352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b5936cc2ee4345b7ad6682dcb8f396" candidate_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718226ccbc2742f3a8429238ae014ae8" is_pre_election: true
I20260811 11:16:16.464136 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b5936cc2ee4345b7ad6682dcb8f396" candidate_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e7592509d4f6ba4608a6f60076edc" is_pre_election: true
I20260811 11:16:16.464285 5037 raft_consensus.cc:2454] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b419b6e7f7f4a6eb9a7853d6a21a7d2 in term 0.
I20260811 11:16:16.464285 5352 raft_consensus.cc:2454] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b419b6e7f7f4a6eb9a7853d6a21a7d2 in term 0.
I20260811 11:16:16.464584 5118 leader_election.cc:302] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, 718226ccbc2742f3a8429238ae014ae8; no voters:
I20260811 11:16:16.464753 5444 raft_consensus.cc:2781] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:16:16.464808 5444 raft_consensus.cc:482] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:16:16.464848 5444 raft_consensus.cc:3037] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:16.465914 5444 raft_consensus.cc:504] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.466094 5444 leader_election.cc:288] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [CANDIDATE]: Term 1 election: Requested vote from peers 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073), 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3:37919)
I20260811 11:16:16.466300 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b5936cc2ee4345b7ad6682dcb8f396" candidate_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e7592509d4f6ba4608a6f60076edc"
I20260811 11:16:16.466336 5352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b5936cc2ee4345b7ad6682dcb8f396" candidate_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718226ccbc2742f3a8429238ae014ae8"
I20260811 11:16:16.466382 5037 raft_consensus.cc:3037] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:16.466493 5352 raft_consensus.cc:3037] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:16.467387 5037 raft_consensus.cc:2454] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b419b6e7f7f4a6eb9a7853d6a21a7d2 in term 1.
I20260811 11:16:16.467638 5119 leader_election.cc:302] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, 980e7592509d4f6ba4608a6f60076edc; no voters:
I20260811 11:16:16.467775 5444 raft_consensus.cc:2781] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:16.467816 5352 raft_consensus.cc:2454] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b419b6e7f7f4a6eb9a7853d6a21a7d2 in term 1.
I20260811 11:16:16.467976 5444 raft_consensus.cc:686] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, State: Running, Role: LEADER
I20260811 11:16:16.468127 5444 consensus_queue.cc:237] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.469120 4857 catalog_manager.cc:5495] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2). New cstate: current_term: 1 leader_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } health_report { overall_health: UNKNOWN } } }
I20260811 11:16:16.485034 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.481833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.28.25.2:44577)
0811 11:16:16.481883 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.481885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.481891 (+ 6us) server_negotiation.cc:407] Connection header received
0811 11:16:16.481930 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.481931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.481957 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.482035 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:16:16.482279 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.482804 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.483911 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.484131 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.484199 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.484450 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:16:16.484580 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:16:16.484663 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.484785 (+ 122us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.484811 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 11:16:16.487516 5037 raft_consensus.cc:1261] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Refusing update from remote peer 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:16.487515 5352 raft_consensus.cc:1261] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Refusing update from remote peer 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:16.488041 5447 consensus_queue.cc:1035] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:16.488209 5444 consensus_queue.cc:1035] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:16.491631 5450 mvcc.cc:205] Tried to move back new op lower bound from 7317286815688929280 to 7317286815614414848. Current Snapshot: MvccSnapshot[applied={T|T < 7317286815688929280 or (T in {7317286815688929280})}]
I20260811 11:16:16.491899 5453 mvcc.cc:205] Tried to move back new op lower bound from 7317286815688929280 to 7317286815614414848. Current Snapshot: MvccSnapshot[applied={T|T < 7317286815688929280 or (T in {7317286815688929280})}]
I20260811 11:16:16.493177 5451 mvcc.cc:205] Tried to move back new op lower bound from 7317286815688929280 to 7317286815614414848. Current Snapshot: MvccSnapshot[applied={T|T < 7317286815688929280}]
I20260811 11:16:16.493489 4858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:16:16.493600 4858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:16:16.494407 4858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79: Sending DeleteTablet for 3 replicas of tablet 92b5936cc2ee4345b7ad6682dcb8f396
I20260811 11:16:16.494831 5017 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b5936cc2ee4345b7ad6682dcb8f396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:16 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:54618
I20260811 11:16:16.494830 5174 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b5936cc2ee4345b7ad6682dcb8f396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:16 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:39820
I20260811 11:16:16.495024 5314 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b5936cc2ee4345b7ad6682dcb8f396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:16 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:59292
I20260811 11:16:16.495362 5461 tablet_replica.cc:330] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: stopping tablet replica
I20260811 11:16:16.495378 4858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 11:16:16.495509 5461 raft_consensus.cc:2229] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Raft consensus shutting down.
W20260811 11:16:16.495551 4858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:16:16.495643 5461 raft_consensus.cc:2258] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:16.495838 5462 tablet_replica.cc:330] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: stopping tablet replica
I20260811 11:16:16.495949 5462 raft_consensus.cc:2229] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:16:16.496060 5462 raft_consensus.cc:2258] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:16.496618 5462 ts_tablet_manager.cc:1905] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:16.497032 5017 tablet_service.cc:1467] Processing CreateTablet for tablet 9dba0430467c4bdc935dda5bad5fac4b (DEFAULT_TABLE table=test-table [id=a3922427e53f4c98bc5f4a1305664c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:16.497040 5174 tablet_service.cc:1467] Processing CreateTablet for tablet 9dba0430467c4bdc935dda5bad5fac4b (DEFAULT_TABLE table=test-table [id=a3922427e53f4c98bc5f4a1305664c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:16.497166 5174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dba0430467c4bdc935dda5bad5fac4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.497169 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dba0430467c4bdc935dda5bad5fac4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.498682 5438 tablet_bootstrap.cc:492] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Bootstrap starting.
I20260811 11:16:16.498929 5461 ts_tablet_manager.cc:1905] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:16.499066 5437 tablet_bootstrap.cc:492] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: Bootstrap starting.
I20260811 11:16:16.499277 5314 tablet_service.cc:1467] Processing CreateTablet for tablet 9dba0430467c4bdc935dda5bad5fac4b (DEFAULT_TABLE table=test-table [id=a3922427e53f4c98bc5f4a1305664c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:16.499387 5438 tablet_bootstrap.cc:654] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.499435 5314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dba0430467c4bdc935dda5bad5fac4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.499518 5437 tablet_bootstrap.cc:654] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.500371 5438 tablet_bootstrap.cc:492] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: No bootstrap required, opened a new log
I20260811 11:16:16.500419 5438 ts_tablet_manager.cc:1397] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:16:16.500627 5438 raft_consensus.cc:348] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.500728 5438 raft_consensus.cc:374] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.500757 5438 raft_consensus.cc:729] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.500809 5438 consensus_queue.cc:260] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.500909 5462 ts_tablet_manager.cc:1918] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:16:16.500931 5438 ts_tablet_manager.cc:1428] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 11:16:16.500984 5462 log.cc:1196] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/92b5936cc2ee4345b7ad6682dcb8f396
I20260811 11:16:16.501260 5441 tablet_bootstrap.cc:492] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8: Bootstrap starting.
I20260811 11:16:16.501287 5462 ts_tablet_manager.cc:1939] T 92b5936cc2ee4345b7ad6682dcb8f396 P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting consensus metadata
I20260811 11:16:16.501762 5441 tablet_bootstrap.cc:654] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.501817 5461 ts_tablet_manager.cc:1918] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:16:16.501899 5461 log.cc:1196] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/92b5936cc2ee4345b7ad6682dcb8f396
I20260811 11:16:16.501984 5437 tablet_bootstrap.cc:492] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: No bootstrap required, opened a new log
I20260811 11:16:16.502025 5437 ts_tablet_manager.cc:1397] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 11:16:16.502184 5461 ts_tablet_manager.cc:1939] T 92b5936cc2ee4345b7ad6682dcb8f396 P 980e7592509d4f6ba4608a6f60076edc: Deleting consensus metadata
I20260811 11:16:16.502197 5437 raft_consensus.cc:348] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.502285 5437 raft_consensus.cc:374] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.502310 5437 raft_consensus.cc:729] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e7592509d4f6ba4608a6f60076edc, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.502288 4840 catalog_manager.cc:4841] TS 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577): tablet 92b5936cc2ee4345b7ad6682dcb8f396 (table test-table [id=92de1498f76b423cabe3f4f6ceb6f744]) successfully deleted
I20260811 11:16:16.502358 5437 consensus_queue.cc:260] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.502446 5437 ts_tablet_manager.cc:1428] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 11:16:16.502640 4838 catalog_manager.cc:4841] TS 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073): tablet 92b5936cc2ee4345b7ad6682dcb8f396 (table test-table [id=92de1498f76b423cabe3f4f6ceb6f744]) successfully deleted
I20260811 11:16:16.502848 5463 tablet_replica.cc:330] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: stopping tablet replica
I20260811 11:16:16.503008 5463 raft_consensus.cc:2229] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:16:16.503165 5463 raft_consensus.cc:2258] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:16.503683 5463 ts_tablet_manager.cc:1905] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:16.505626 5463 ts_tablet_manager.cc:1918] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:16:16.505708 5463 log.cc:1196] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/92b5936cc2ee4345b7ad6682dcb8f396
I20260811 11:16:16.505982 5463 ts_tablet_manager.cc:1939] T 92b5936cc2ee4345b7ad6682dcb8f396 P 718226ccbc2742f3a8429238ae014ae8: Deleting consensus metadata
I20260811 11:16:16.506487 4837 catalog_manager.cc:4841] TS 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3:37919): tablet 92b5936cc2ee4345b7ad6682dcb8f396 (table test-table [id=92de1498f76b423cabe3f4f6ceb6f744]) successfully deleted
I20260811 11:16:16.507467 5441 tablet_bootstrap.cc:492] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8: No bootstrap required, opened a new log
I20260811 11:16:16.507524 5441 ts_tablet_manager.cc:1397] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260811 11:16:16.507690 5441 raft_consensus.cc:348] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.507767 5441 raft_consensus.cc:374] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.507787 5441 raft_consensus.cc:729] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718226ccbc2742f3a8429238ae014ae8, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.507840 5441 consensus_queue.cc:260] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.507936 5441 ts_tablet_manager.cc:1428] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260811 11:16:16.513090 5093 tablet.cc:2369] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:16:16.548303 5445 raft_consensus.cc:482] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:16:16.548480 5445 raft_consensus.cc:504] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.548885 5445 leader_election.cc:288] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073), 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577)
I20260811 11:16:16.552925 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.549087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:44577 (local address 127.28.25.3:41967)
0811 11:16:16.549395 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.549405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.549562 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.549769 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.549770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.549772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:16.549873 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.549874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.550707 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.550708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.551788 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.551793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.552586 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.552588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.552675 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.552680 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.552747 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.552797 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":122,"thread_start_us":106,"threads_started":1}
I20260811 11:16:16.552938 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.549186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:41967 (local address 127.28.25.2:44577)
0811 11:16:16.549360 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.549362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.549479 (+ 117us) server_negotiation.cc:407] Connection header received
0811 11:16:16.549662 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.549664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.549673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.549703 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:16.549967 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.550607 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.551885 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.552475 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.552565 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.552578 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.552777 (+ 199us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.552825 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260811 11:16:16.553032 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.549077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:38073 (local address 127.28.25.3:34765)
0811 11:16:16.549157 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.549170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.549224 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.549446 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.549446 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.549469 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:16.549548 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.549551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.550508 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.550509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.551897 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.551901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.552755 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.552757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.552837 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.552841 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.552894 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.552954 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 11:16:16.553371 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.549167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:34765 (local address 127.28.25.1:38073)
0811 11:16:16.549211 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.549214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.549225 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:16:16.549316 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.549317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.549329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.549384 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:16.549642 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.550388 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.552008 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.552607 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.552685 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.552694 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.553222 (+ 528us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.553293 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:16:16.553493 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dba0430467c4bdc935dda5bad5fac4b" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" is_pre_election: true
I20260811 11:16:16.553565 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dba0430467c4bdc935dda5bad5fac4b" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e7592509d4f6ba4608a6f60076edc" is_pre_election: true
I20260811 11:16:16.553653 5037 raft_consensus.cc:2454] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 0.
I20260811 11:16:16.553659 5195 raft_consensus.cc:2454] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 0.
I20260811 11:16:16.553908 5277 leader_election.cc:302] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, 718226ccbc2742f3a8429238ae014ae8; no voters:
I20260811 11:16:16.554085 5445 raft_consensus.cc:2781] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:16:16.554124 5445 raft_consensus.cc:482] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:16:16.554145 5445 raft_consensus.cc:3037] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:16.555133 5445 raft_consensus.cc:504] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.555308 5445 leader_election.cc:288] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 election: Requested vote from peers 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073), 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577)
I20260811 11:16:16.555559 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dba0430467c4bdc935dda5bad5fac4b" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e7592509d4f6ba4608a6f60076edc"
I20260811 11:16:16.555650 5037 raft_consensus.cc:3037] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:16.556602 5037 raft_consensus.cc:2454] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 1.
I20260811 11:16:16.556749 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dba0430467c4bdc935dda5bad5fac4b" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2"
I20260811 11:16:16.556836 5195 raft_consensus.cc:3037] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:16.556847 5275 leader_election.cc:302] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718226ccbc2742f3a8429238ae014ae8, 980e7592509d4f6ba4608a6f60076edc; no voters:
I20260811 11:16:16.557013 5445 raft_consensus.cc:2781] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:16.557156 5445 raft_consensus.cc:686] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 1 LEADER]: Becoming Leader. State: Replica: 718226ccbc2742f3a8429238ae014ae8, State: Running, Role: LEADER
I20260811 11:16:16.557241 5445 consensus_queue.cc:237] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.557674 5195 raft_consensus.cc:2454] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 1.
I20260811 11:16:16.558127 4857 catalog_manager.cc:5495] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3). New cstate: current_term: 1 leader_uuid: "718226ccbc2742f3a8429238ae014ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } health_report { overall_health: HEALTHY } } }
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446975, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.62@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:16:16.596235 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.588174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.28.25.62:42787)
0811 11:16:16.588232 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.588234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.588335 (+ 101us) server_negotiation.cc:407] Connection header received
0811 11:16:16.588453 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.588454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.588466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.588545 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.589245 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.589768 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.590592 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.590786 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.590879 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.593273 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:16.593274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:16.593275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:16.593283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:16.594817 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.595304 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.595305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.595306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.595368 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.595591 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.595592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.595593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.595718 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:16.595796 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.595986 (+ 190us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.596104 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260811 11:16:16.596338 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.587987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42787 (local address 127.0.0.1:40720)
0811 11:16:16.588214 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.588247 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.588353 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.588551 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.588552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.589076 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:16.589137 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.589140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.589864 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.589865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.590493 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.590495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.590880 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.590881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.591007 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.591635 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:16.591640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:16.593165 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:16.594929 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.594943 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.594946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.595218 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.595457 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:16.595458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:16.595458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:16.595525 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:16.595800 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:16.595813 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:16.595863 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.595957 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.596175 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":93,"thread_start_us":117,"threads_started":1}
I20260811 11:16:16.600754 4857 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:40720:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 11:16:16.601081 4857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:16:16.602622 5174 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4b362ffcf94eb28539feaa0484e40d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91bb66149ffd4945b248c74711ea562d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:16:16.602605 5017 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4b362ffcf94eb28539feaa0484e40d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91bb66149ffd4945b248c74711ea562d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:16:16.602816 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4b362ffcf94eb28539feaa0484e40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.602841 5174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4b362ffcf94eb28539feaa0484e40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.604761 5438 tablet_bootstrap.cc:492] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Bootstrap starting.
I20260811 11:16:16.604864 5437 tablet_bootstrap.cc:492] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: Bootstrap starting.
I20260811 11:16:16.605293 5437 tablet_bootstrap.cc:654] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.605309 5438 tablet_bootstrap.cc:654] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.605755 5314 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4b362ffcf94eb28539feaa0484e40d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91bb66149ffd4945b248c74711ea562d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:16:16.605933 5314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4b362ffcf94eb28539feaa0484e40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.607769 5441 tablet_bootstrap.cc:492] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: Bootstrap starting.
I20260811 11:16:16.608232 5437 tablet_bootstrap.cc:492] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: No bootstrap required, opened a new log
I20260811 11:16:16.608286 5437 ts_tablet_manager.cc:1397] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 11:16:16.608294 5441 tablet_bootstrap.cc:654] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.608441 5437 raft_consensus.cc:348] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.608505 5437 raft_consensus.cc:374] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.608520 5437 raft_consensus.cc:729] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e7592509d4f6ba4608a6f60076edc, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.608593 5437 consensus_queue.cc:260] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.608729 5437 ts_tablet_manager.cc:1428] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:16.608816 5443 tablet_replica.cc:439] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } } }
I20260811 11:16:16.608942 5443 tablet_replica.cc:442] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:16:16.609272 5438 tablet_bootstrap.cc:492] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: No bootstrap required, opened a new log
I20260811 11:16:16.609350 5438 ts_tablet_manager.cc:1397] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260811 11:16:16.609557 5438 raft_consensus.cc:348] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.609686 5438 raft_consensus.cc:374] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.609692 5441 tablet_bootstrap.cc:492] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: No bootstrap required, opened a new log
I20260811 11:16:16.609716 5438 raft_consensus.cc:729] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.609747 5441 ts_tablet_manager.cc:1397] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260811 11:16:16.609769 5438 consensus_queue.cc:260] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.609912 5438 ts_tablet_manager.cc:1428] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:16.609929 5447 tablet_replica.cc:439] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } } }
I20260811 11:16:16.610042 5447 tablet_replica.cc:442] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:16:16.610131 5441 raft_consensus.cc:348] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.610219 5441 raft_consensus.cc:374] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.610258 5441 raft_consensus.cc:729] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718226ccbc2742f3a8429238ae014ae8, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.610318 5441 consensus_queue.cc:260] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.610412 5441 ts_tablet_manager.cc:1428] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 11:16:16.610754 5445 tablet_replica.cc:439] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } } }
I20260811 11:16:16.610858 5445 tablet_replica.cc:442] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: This TxnStatusTablet replica's current role is: FOLLOWER
W20260811 11:16:16.632970 5403 tablet.cc:2369] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:16:16.696697 5249 tablet.cc:2369] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:16:16.754210 5445 raft_consensus.cc:482] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:16:16.754323 5445 raft_consensus.cc:504] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.754501 5445 leader_election.cc:288] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073), 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577)
I20260811 11:16:16.754846 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4b362ffcf94eb28539feaa0484e40d" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" is_pre_election: true
I20260811 11:16:16.754837 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4b362ffcf94eb28539feaa0484e40d" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e7592509d4f6ba4608a6f60076edc" is_pre_election: true
I20260811 11:16:16.754952 5037 raft_consensus.cc:2454] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 0.
I20260811 11:16:16.754952 5195 raft_consensus.cc:2454] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 0.
I20260811 11:16:16.755174 5277 leader_election.cc:302] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, 718226ccbc2742f3a8429238ae014ae8; no voters:
I20260811 11:16:16.755326 5445 raft_consensus.cc:2781] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:16:16.755420 5445 raft_consensus.cc:482] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:16:16.755456 5445 raft_consensus.cc:3037] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:16.756434 5445 raft_consensus.cc:504] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.756615 5445 leader_election.cc:288] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 election: Requested vote from peers 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073), 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577)
I20260811 11:16:16.756825 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4b362ffcf94eb28539feaa0484e40d" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e7592509d4f6ba4608a6f60076edc"
I20260811 11:16:16.756827 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4b362ffcf94eb28539feaa0484e40d" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2"
I20260811 11:16:16.756914 5037 raft_consensus.cc:3037] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:16.756940 5195 raft_consensus.cc:3037] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:16.757733 5037 raft_consensus.cc:2454] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 1.
I20260811 11:16:16.757848 5195 raft_consensus.cc:2454] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 1.
I20260811 11:16:16.757961 5275 leader_election.cc:302] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718226ccbc2742f3a8429238ae014ae8, 980e7592509d4f6ba4608a6f60076edc; no voters:
I20260811 11:16:16.758060 5445 raft_consensus.cc:2781] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:16.758111 5445 raft_consensus.cc:686] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [term 1 LEADER]: Becoming Leader. State: Replica: 718226ccbc2742f3a8429238ae014ae8, State: Running, Role: LEADER
I20260811 11:16:16.758160 5445 consensus_queue.cc:237] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.758518 5468 tablet_replica.cc:439] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: TxnStatusTablet state changed. Reason: New leader 718226ccbc2742f3a8429238ae014ae8. Latest consensus state: current_term: 1 leader_uuid: "718226ccbc2742f3a8429238ae014ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } } }
I20260811 11:16:16.758610 5468 tablet_replica.cc:442] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:16:16.758837 5485 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 11:16:16.758940 5485 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 11:16:16.759052 4857 catalog_manager.cc:5495] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3). New cstate: current_term: 1 leader_uuid: "718226ccbc2742f3a8429238ae014ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } health_report { overall_health: HEALTHY } } }
I20260811 11:16:16.795004 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.791852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.28.25.3:37919)
0811 11:16:16.791941 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.791944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.791957 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:16:16.792020 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.792021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.792046 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.792123 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:16:16.792276 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.793050 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.793870 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.794186 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.794270 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.794481 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:16:16.794632 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:16:16.794732 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.794866 (+ 134us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.794895 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 11:16:16.795063 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.791755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:37919 (local address 127.0.0.1:59308)
0811 11:16:16.791859 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.791874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.791933 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.792139 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.792140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.792141 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:16:16.792193 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.792195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.793161 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.793162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.793767 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.793769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.794273 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.794274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.794332 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.794355 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:16:16.794758 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:16:16.794775 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.794849 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.794906 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 11:16:16.796578 5037 raft_consensus.cc:1261] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Refusing update from remote peer 718226ccbc2742f3a8429238ae014ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:16.796646 5195 raft_consensus.cc:1261] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Refusing update from remote peer 718226ccbc2742f3a8429238ae014ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:16.797039 5445 consensus_queue.cc:1035] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:16.797205 5468 consensus_queue.cc:1035] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:16.797874 5443 tablet_replica.cc:439] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: TxnStatusTablet state changed. Reason: New leader 718226ccbc2742f3a8429238ae014ae8. Latest consensus state: current_term: 1 leader_uuid: "718226ccbc2742f3a8429238ae014ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } } }
I20260811 11:16:16.797979 5443 tablet_replica.cc:442] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:16:16.797991 5447 tablet_replica.cc:439] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: TxnStatusTablet state changed. Reason: New leader 718226ccbc2742f3a8429238ae014ae8. Latest consensus state: current_term: 1 leader_uuid: "718226ccbc2742f3a8429238ae014ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } } }
I20260811 11:16:16.798097 5447 tablet_replica.cc:442] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:16:16.799202 5443 tablet_replica.cc:439] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "718226ccbc2742f3a8429238ae014ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } } }
I20260811 11:16:16.799434 5443 tablet_replica.cc:442] T 3c4b362ffcf94eb28539feaa0484e40d P 980e7592509d4f6ba4608a6f60076edc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:16:16.798995 5468 tablet_replica.cc:439] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "718226ccbc2742f3a8429238ae014ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } } }
I20260811 11:16:16.800328 5468 tablet_replica.cc:442] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:16:16.800028 5445 tablet_replica.cc:439] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "718226ccbc2742f3a8429238ae014ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } } }
I20260811 11:16:16.800484 5445 tablet_replica.cc:442] T 3c4b362ffcf94eb28539feaa0484e40d P 718226ccbc2742f3a8429238ae014ae8: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:16:16.800542 5447 tablet_replica.cc:439] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "718226ccbc2742f3a8429238ae014ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } } }
I20260811 11:16:16.800634 5447 tablet_replica.cc:442] T 3c4b362ffcf94eb28539feaa0484e40d P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:16:16.804037 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.801226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.28.25.3:37919)
0811 11:16:16.801366 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.801369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.801377 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:16:16.801401 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.801402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.801415 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.801464 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:16:16.801625 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.802265 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.803194 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.803425 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.803519 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.803687 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:16:16.803735 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:16:16.803823 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.803898 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.803923 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260811 11:16:16.810699 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.807305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:37919 (local address 127.28.25.3:35063)
0811 11:16:16.807414 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:16.807429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:16.807488 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:16.807624 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:16.807625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:16.807626 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:16:16.807979 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.807982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.808961 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.808962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.809610 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:16.809612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.810118 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.810119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:16.810183 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.810207 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:16:16.810490 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:16:16.810493 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:16:16.810549 (+ 56us) client_negotiation.cc:240] Negotiation successful
0811 11:16:16.810597 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 11:16:16.810709 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.807423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:35063 (local address 127.28.25.3:37919)
0811 11:16:16.807495 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.807497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.807511 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:16:16.807537 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.807538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.807550 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.807613 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:16:16.808074 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.808852 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.809709 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.809960 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.810037 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.810344 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:16:16.810383 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:16:16.810444 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.810580 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.810602 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:16:16.814291 5195 raft_consensus.cc:1261] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Refusing update from remote peer 718226ccbc2742f3a8429238ae014ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:16.814332 5037 raft_consensus.cc:1261] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Refusing update from remote peer 718226ccbc2742f3a8429238ae014ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:16.814558 5468 consensus_queue.cc:1035] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:16.814777 5468 consensus_queue.cc:1035] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:16.830602 4858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:16:16.830742 4858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:16:16.831599 4858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79: Sending DeleteTablet for 3 replicas of tablet 9dba0430467c4bdc935dda5bad5fac4b
I20260811 11:16:16.831944 5017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dba0430467c4bdc935dda5bad5fac4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:16 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:54618
I20260811 11:16:16.832082 5461 tablet_replica.cc:330] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: stopping tablet replica
I20260811 11:16:16.832072 5174 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dba0430467c4bdc935dda5bad5fac4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:16 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:39820
I20260811 11:16:16.832137 5461 raft_consensus.cc:2229] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:16:16.832183 5461 raft_consensus.cc:2258] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:16.832280 5462 tablet_replica.cc:330] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: stopping tablet replica
I20260811 11:16:16.832352 5462 raft_consensus.cc:2229] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:16:16.832401 5462 raft_consensus.cc:2258] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:16.832598 5330 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dba0430467c4bdc935dda5bad5fac4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:16 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:59292
I20260811 11:16:16.832626 5461 ts_tablet_manager.cc:1905] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:16.832710 5463 tablet_replica.cc:330] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8: stopping tablet replica
I20260811 11:16:16.832768 5463 raft_consensus.cc:2229] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:16:16.832880 5463 raft_consensus.cc:2258] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:16.833196 5462 ts_tablet_manager.cc:1905] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:16.834553 5463 ts_tablet_manager.cc:1905] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:16.834702 5461 ts_tablet_manager.cc:1918] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:16:16.834800 5461 log.cc:1196] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/9dba0430467c4bdc935dda5bad5fac4b
I20260811 11:16:16.835085 5461 ts_tablet_manager.cc:1939] T 9dba0430467c4bdc935dda5bad5fac4b P 980e7592509d4f6ba4608a6f60076edc: Deleting consensus metadata
I20260811 11:16:16.835870 4838 catalog_manager.cc:4841] TS 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073): tablet 9dba0430467c4bdc935dda5bad5fac4b (table test-table [id=a3922427e53f4c98bc5f4a1305664c06]) successfully deleted
I20260811 11:16:16.836819 5463 ts_tablet_manager.cc:1918] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:16:16.836815 5462 ts_tablet_manager.cc:1918] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:16:16.836889 5463 log.cc:1196] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/9dba0430467c4bdc935dda5bad5fac4b
I20260811 11:16:16.836890 5462 log.cc:1196] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/9dba0430467c4bdc935dda5bad5fac4b
I20260811 11:16:16.837121 5463 ts_tablet_manager.cc:1939] T 9dba0430467c4bdc935dda5bad5fac4b P 718226ccbc2742f3a8429238ae014ae8: Deleting consensus metadata
I20260811 11:16:16.837124 5462 ts_tablet_manager.cc:1939] T 9dba0430467c4bdc935dda5bad5fac4b P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting consensus metadata
I20260811 11:16:16.837536 4840 catalog_manager.cc:4841] TS 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577): tablet 9dba0430467c4bdc935dda5bad5fac4b (table test-table [id=a3922427e53f4c98bc5f4a1305664c06]) successfully deleted
I20260811 11:16:16.838636 4837 catalog_manager.cc:4841] TS 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3:37919): tablet 9dba0430467c4bdc935dda5bad5fac4b (table test-table [id=a3922427e53f4c98bc5f4a1305664c06]) successfully deleted
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 11:16:16.863261 4933 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:16.857173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.28.25.62:42787)
0811 11:16:16.857226 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.857228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.857432 (+ 204us) server_negotiation.cc:407] Connection header received
0811 11:16:16.857535 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.857536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.857554 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.857624 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.858289 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.858803 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.859702 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.859916 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.860005 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.863185 (+ 3180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40728: BlockingRecv error: recv got EOF from 127.0.0.1:40728 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 11 11:16:16 dist-test-slave-2w3w krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM
I20260811 11:16:16.873387 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:16.864154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.28.25.62:42787)
0811 11:16:16.864234 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:16.864236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:16.864311 (+ 75us) server_negotiation.cc:407] Connection header received
0811 11:16:16.864372 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:16.864373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:16.864385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:16.864464 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:16.864973 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.865780 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.866550 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:16.866864 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:16.866953 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:16.869788 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:16.869790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:16.869790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:16.869801 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:16.871875 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.872348 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.872350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.872350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.872428 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:16.872669 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:16.872670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:16.872671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:16.872845 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:16.872966 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:16.873166 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 11:16:16.873285 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:16:16.875443 4858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:16:16.875672 4858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:16:16.877303 5017 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca (DEFAULT_TABLE table=test-table [id=e8e806971d2f4dc7af1fd2cc6461717c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:16.877350 5174 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca (DEFAULT_TABLE table=test-table [id=e8e806971d2f4dc7af1fd2cc6461717c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:16.877512 5174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.877513 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.879067 5437 tablet_bootstrap.cc:492] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc: Bootstrap starting.
I20260811 11:16:16.879153 5438 tablet_bootstrap.cc:492] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Bootstrap starting.
I20260811 11:16:16.879562 5437 tablet_bootstrap.cc:654] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.879563 5438 tablet_bootstrap.cc:654] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.879925 5330 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca (DEFAULT_TABLE table=test-table [id=e8e806971d2f4dc7af1fd2cc6461717c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:16.880062 5330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:16.881604 5441 tablet_bootstrap.cc:492] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8: Bootstrap starting.
I20260811 11:16:16.882128 5438 tablet_bootstrap.cc:492] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: No bootstrap required, opened a new log
I20260811 11:16:16.882140 5441 tablet_bootstrap.cc:654] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:16.882195 5438 ts_tablet_manager.cc:1397] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 11:16:16.882356 5438 raft_consensus.cc:348] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.882449 5438 raft_consensus.cc:374] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.882470 5438 raft_consensus.cc:729] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.882514 5438 consensus_queue.cc:260] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.882607 5438 ts_tablet_manager.cc:1428] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 11:16:16.883248 5437 tablet_bootstrap.cc:492] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc: No bootstrap required, opened a new log
I20260811 11:16:16.883319 5437 ts_tablet_manager.cc:1397] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260811 11:16:16.883503 5437 raft_consensus.cc:348] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.883602 5437 raft_consensus.cc:374] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.883638 5437 raft_consensus.cc:729] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e7592509d4f6ba4608a6f60076edc, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.883700 5437 consensus_queue.cc:260] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.883781 5441 tablet_bootstrap.cc:492] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8: No bootstrap required, opened a new log
I20260811 11:16:16.883828 5437 ts_tablet_manager.cc:1428] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 11:16:16.883828 5441 ts_tablet_manager.cc:1397] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260811 11:16:16.884018 5441 raft_consensus.cc:348] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.884085 5441 raft_consensus.cc:374] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:16.884099 5441 raft_consensus.cc:729] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718226ccbc2742f3a8429238ae014ae8, State: Initialized, Role: FOLLOWER
I20260811 11:16:16.884162 5441 consensus_queue.cc:260] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:16.884270 5441 ts_tablet_manager.cc:1428] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:17.210274 5491 raft_consensus.cc:482] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:16:17.210459 5491 raft_consensus.cc:504] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.210665 5491 leader_election.cc:288] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073), 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577)
I20260811 11:16:17.210929 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea6c58a3c8144ed8b13f2eac3eb7cca" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" is_pre_election: true
I20260811 11:16:17.210945 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea6c58a3c8144ed8b13f2eac3eb7cca" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e7592509d4f6ba4608a6f60076edc" is_pre_election: true
I20260811 11:16:17.211103 5195 raft_consensus.cc:2454] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 0.
I20260811 11:16:17.211119 5037 raft_consensus.cc:2454] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 0.
I20260811 11:16:17.211328 5277 leader_election.cc:302] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, 718226ccbc2742f3a8429238ae014ae8; no voters:
I20260811 11:16:17.211464 5491 raft_consensus.cc:2781] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:16:17.211553 5491 raft_consensus.cc:482] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:16:17.211589 5491 raft_consensus.cc:3037] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:17.212507 5491 raft_consensus.cc:504] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.212728 5491 leader_election.cc:288] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 election: Requested vote from peers 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073), 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577)
I20260811 11:16:17.212954 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea6c58a3c8144ed8b13f2eac3eb7cca" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e7592509d4f6ba4608a6f60076edc"
I20260811 11:16:17.212989 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea6c58a3c8144ed8b13f2eac3eb7cca" candidate_uuid: "718226ccbc2742f3a8429238ae014ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2"
I20260811 11:16:17.213075 5195 raft_consensus.cc:3037] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:17.213075 5037 raft_consensus.cc:3037] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:17.214004 5037 raft_consensus.cc:2454] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 1.
I20260811 11:16:17.214004 5195 raft_consensus.cc:2454] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718226ccbc2742f3a8429238ae014ae8 in term 1.
I20260811 11:16:17.214250 5277 leader_election.cc:302] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, 718226ccbc2742f3a8429238ae014ae8; no voters:
I20260811 11:16:17.214407 5491 raft_consensus.cc:2781] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:17.214489 5491 raft_consensus.cc:686] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 1 LEADER]: Becoming Leader. State: Replica: 718226ccbc2742f3a8429238ae014ae8, State: Running, Role: LEADER
I20260811 11:16:17.214553 5491 consensus_queue.cc:237] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.215359 4858 catalog_manager.cc:5495] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3). New cstate: current_term: 1 leader_uuid: "718226ccbc2742f3a8429238ae014ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } health_report { overall_health: HEALTHY } } }
I20260811 11:16:17.243393 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:17.240777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.28.25.3:37919)
0811 11:16:17.240863 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:17.240865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:17.240881 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:16:17.240908 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:17.240908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:17.240918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:17.240980 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:16:17.241163 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.241684 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.242531 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.242727 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.242788 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:17.243099 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:16:17.243134 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:16:17.243190 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:17.243240 (+ 50us) server_negotiation.cc:299] Negotiation successful
0811 11:16:17.243260 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 11:16:17.244455 5195 raft_consensus.cc:1261] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Refusing update from remote peer 718226ccbc2742f3a8429238ae014ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:17.244484 5037 raft_consensus.cc:1261] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Refusing update from remote peer 718226ccbc2742f3a8429238ae014ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:17.244756 5491 consensus_queue.cc:1035] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:17.244901 5445 consensus_queue.cc:1035] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:17.248106 4857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:16:17.248203 4857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:16:17.248916 4857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79: Sending DeleteTablet for 3 replicas of tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca
I20260811 11:16:17.249209 5174 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:17 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:39820
I20260811 11:16:17.249343 5330 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:17 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:59292
I20260811 11:16:17.249377 5017 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:17 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:54618
I20260811 11:16:17.249444 5463 tablet_replica.cc:330] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8: stopping tablet replica
I20260811 11:16:17.249451 5462 tablet_replica.cc:330] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: stopping tablet replica
I20260811 11:16:17.249504 5463 raft_consensus.cc:2229] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:16:17.249512 5462 raft_consensus.cc:2229] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:16:17.249560 5462 raft_consensus.cc:2258] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:17.249586 5463 raft_consensus.cc:2258] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:17.249650 5461 tablet_replica.cc:330] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc: stopping tablet replica
I20260811 11:16:17.249792 5461 raft_consensus.cc:2229] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:16:17.249866 5461 raft_consensus.cc:2258] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:17.250203 5461 ts_tablet_manager.cc:1905] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:17.250221 5462 ts_tablet_manager.cc:1905] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:17.251722 5463 ts_tablet_manager.cc:1905] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:17.252127 5462 ts_tablet_manager.cc:1918] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:16:17.252151 5461 ts_tablet_manager.cc:1918] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:16:17.252261 5462 log.cc:1196] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/8ea6c58a3c8144ed8b13f2eac3eb7cca
I20260811 11:16:17.252282 5461 log.cc:1196] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/8ea6c58a3c8144ed8b13f2eac3eb7cca
I20260811 11:16:17.252542 5461 ts_tablet_manager.cc:1939] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 980e7592509d4f6ba4608a6f60076edc: Deleting consensus metadata
I20260811 11:16:17.252540 5462 ts_tablet_manager.cc:1939] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting consensus metadata
I20260811 11:16:17.253005 4840 catalog_manager.cc:4841] TS 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577): tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca (table test-table [id=e8e806971d2f4dc7af1fd2cc6461717c]) successfully deleted
I20260811 11:16:17.253078 4838 catalog_manager.cc:4841] TS 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073): tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca (table test-table [id=e8e806971d2f4dc7af1fd2cc6461717c]) successfully deleted
I20260811 11:16:17.253845 5463 ts_tablet_manager.cc:1918] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:16:17.253922 5463 log.cc:1196] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/8ea6c58a3c8144ed8b13f2eac3eb7cca
I20260811 11:16:17.254168 5463 ts_tablet_manager.cc:1939] T 8ea6c58a3c8144ed8b13f2eac3eb7cca P 718226ccbc2742f3a8429238ae014ae8: Deleting consensus metadata
I20260811 11:16:17.254045 4857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:16:17.254346 4857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:16:17.254663 4837 catalog_manager.cc:4841] TS 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3:37919): tablet 8ea6c58a3c8144ed8b13f2eac3eb7cca (table test-table [id=e8e806971d2f4dc7af1fd2cc6461717c]) successfully deleted
I20260811 11:16:17.256049 5330 tablet_service.cc:1467] Processing CreateTablet for tablet 622b5d392fa0419abeb6d9cc70ad152c (DEFAULT_TABLE table=test-table [id=e8bf748f6af44cd9b8a83e9f9283ea36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:17.256088 5017 tablet_service.cc:1467] Processing CreateTablet for tablet 622b5d392fa0419abeb6d9cc70ad152c (DEFAULT_TABLE table=test-table [id=e8bf748f6af44cd9b8a83e9f9283ea36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:17.256206 5330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622b5d392fa0419abeb6d9cc70ad152c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:17.256254 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622b5d392fa0419abeb6d9cc70ad152c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:17.257596 5441 tablet_bootstrap.cc:492] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8: Bootstrap starting.
I20260811 11:16:17.257668 5174 tablet_service.cc:1467] Processing CreateTablet for tablet 622b5d392fa0419abeb6d9cc70ad152c (DEFAULT_TABLE table=test-table [id=e8bf748f6af44cd9b8a83e9f9283ea36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:17.257814 5174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622b5d392fa0419abeb6d9cc70ad152c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:17.257851 5437 tablet_bootstrap.cc:492] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc: Bootstrap starting.
I20260811 11:16:17.257994 5441 tablet_bootstrap.cc:654] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:17.258247 5437 tablet_bootstrap.cc:654] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:17.258924 5437 tablet_bootstrap.cc:492] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc: No bootstrap required, opened a new log
I20260811 11:16:17.259028 5437 ts_tablet_manager.cc:1397] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:17.259235 5437 raft_consensus.cc:348] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.259366 5437 raft_consensus.cc:374] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:17.259392 5437 raft_consensus.cc:729] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e7592509d4f6ba4608a6f60076edc, State: Initialized, Role: FOLLOWER
I20260811 11:16:17.259394 5438 tablet_bootstrap.cc:492] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Bootstrap starting.
I20260811 11:16:17.259443 5437 consensus_queue.cc:260] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.259564 5437 ts_tablet_manager.cc:1428] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:17.259752 5438 tablet_bootstrap.cc:654] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:17.260200 5441 tablet_bootstrap.cc:492] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8: No bootstrap required, opened a new log
I20260811 11:16:17.260268 5441 ts_tablet_manager.cc:1397] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260811 11:16:17.260397 5441 raft_consensus.cc:348] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.260456 5441 raft_consensus.cc:374] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:17.260470 5441 raft_consensus.cc:729] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718226ccbc2742f3a8429238ae014ae8, State: Initialized, Role: FOLLOWER
I20260811 11:16:17.260536 5441 consensus_queue.cc:260] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.260643 5441 ts_tablet_manager.cc:1428] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:17.260885 5438 tablet_bootstrap.cc:492] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: No bootstrap required, opened a new log
I20260811 11:16:17.260933 5438 ts_tablet_manager.cc:1397] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:16:17.261049 5438 raft_consensus.cc:348] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.261134 5438 raft_consensus.cc:374] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:17.261158 5438 raft_consensus.cc:729] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, State: Initialized, Role: FOLLOWER
I20260811 11:16:17.261205 5438 consensus_queue.cc:260] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.261320 5438 ts_tablet_manager.cc:1428] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:17.353685 5447 raft_consensus.cc:482] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:16:17.353861 5447 raft_consensus.cc:504] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.354091 5447 leader_election.cc:288] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073), 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3:37919)
I20260811 11:16:17.354398 5352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622b5d392fa0419abeb6d9cc70ad152c" candidate_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718226ccbc2742f3a8429238ae014ae8" is_pre_election: true
I20260811 11:16:17.354422 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622b5d392fa0419abeb6d9cc70ad152c" candidate_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e7592509d4f6ba4608a6f60076edc" is_pre_election: true
I20260811 11:16:17.354521 5352 raft_consensus.cc:2454] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b419b6e7f7f4a6eb9a7853d6a21a7d2 in term 0.
I20260811 11:16:17.354521 5037 raft_consensus.cc:2454] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b419b6e7f7f4a6eb9a7853d6a21a7d2 in term 0.
I20260811 11:16:17.354676 5119 leader_election.cc:302] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, 980e7592509d4f6ba4608a6f60076edc; no voters:
I20260811 11:16:17.354902 5447 raft_consensus.cc:2781] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:16:17.354986 5447 raft_consensus.cc:482] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:16:17.355018 5447 raft_consensus.cc:3037] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:17.355866 5447 raft_consensus.cc:504] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.356036 5447 leader_election.cc:288] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [CANDIDATE]: Term 1 election: Requested vote from peers 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073), 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3:37919)
I20260811 11:16:17.356256 5352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622b5d392fa0419abeb6d9cc70ad152c" candidate_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718226ccbc2742f3a8429238ae014ae8"
I20260811 11:16:17.356343 5352 raft_consensus.cc:3037] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:17.356284 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622b5d392fa0419abeb6d9cc70ad152c" candidate_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e7592509d4f6ba4608a6f60076edc"
I20260811 11:16:17.356400 5037 raft_consensus.cc:3037] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:17.357228 5037 raft_consensus.cc:2454] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b419b6e7f7f4a6eb9a7853d6a21a7d2 in term 1.
I20260811 11:16:17.357292 5352 raft_consensus.cc:2454] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b419b6e7f7f4a6eb9a7853d6a21a7d2 in term 1.
I20260811 11:16:17.357497 5118 leader_election.cc:302] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, 718226ccbc2742f3a8429238ae014ae8; no voters:
I20260811 11:16:17.357585 5447 raft_consensus.cc:2781] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:17.357698 5447 raft_consensus.cc:686] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4b419b6e7f7f4a6eb9a7853d6a21a7d2, State: Running, Role: LEADER
I20260811 11:16:17.357754 5447 consensus_queue.cc:237] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } }
I20260811 11:16:17.358635 4858 catalog_manager.cc:5495] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2). New cstate: current_term: 1 leader_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b419b6e7f7f4a6eb9a7853d6a21a7d2" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 } health_report { overall_health: UNKNOWN } } }
I20260811 11:16:17.382990 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:17.379964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.28.25.2:44577)
0811 11:16:17.380145 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:17.380147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:17.380157 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:16:17.380180 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:17.380181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:17.380194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:17.380270 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:16:17.380466 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.381013 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.382073 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.382313 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.382385 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:17.382571 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:16:17.382605 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:16:17.382649 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:17.382722 (+ 73us) server_negotiation.cc:299] Negotiation successful
0811 11:16:17.382745 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260811 11:16:17.391063 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:17.388194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:59049 (local address 127.28.25.3:37919)
0811 11:16:17.388299 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:17.388301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:17.388346 (+ 45us) server_negotiation.cc:407] Connection header received
0811 11:16:17.388428 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:17.388428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:17.388438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:17.388502 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:16:17.388605 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.389200 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.390174 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.390379 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.390440 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:17.390680 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:16:17.390712 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:16:17.390759 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:17.390921 (+ 162us) server_negotiation.cc:299] Negotiation successful
0811 11:16:17.390941 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 11:16:17.391099 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:17.388102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:37919 (local address 127.28.25.2:59049)
0811 11:16:17.388284 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:17.388296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:17.388363 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:17.388507 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:17.388507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:17.388509 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:16:17.388559 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:17.388560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.389282 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.389283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:17.390024 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:17.390027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.390497 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.390499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:17.390563 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:17.390582 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:16:17.390847 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:16:17.390850 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:16:17.390903 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 11:16:17.391014 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260811 11:16:17.393520 5037 raft_consensus.cc:1261] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Refusing update from remote peer 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:17.393510 5352 raft_consensus.cc:1261] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Refusing update from remote peer 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:17.393760 5524 consensus_queue.cc:1035] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "718226ccbc2742f3a8429238ae014ae8" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 37919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:17.393901 5447 consensus_queue.cc:1035] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e7592509d4f6ba4608a6f60076edc" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:17.401906 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:17.399133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:41389 (local address 127.28.25.2:44577)
0811 11:16:17.399179 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:17.399182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:17.399308 (+ 126us) server_negotiation.cc:407] Connection header received
0811 11:16:17.399349 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:17.399349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:17.399363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:17.399410 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:16:17.399496 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.400042 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.400975 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.401347 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.401456 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:17.401629 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:16:17.401667 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:16:17.401717 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:17.401795 (+ 78us) server_negotiation.cc:299] Negotiation successful
0811 11:16:17.401820 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 11:16:17.401919 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:17.399048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:44577 (local address 127.28.25.3:41389)
0811 11:16:17.399233 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:17.399244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:17.399305 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:17.399412 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:17.399412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:17.399414 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:16:17.399459 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:17.399460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.400120 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.400121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:17.400888 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:17.400891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:17.401457 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:17.401458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:17.401524 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:17.401530 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:16:17.401728 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:16:17.401730 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 11:16:17.401790 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 11:16:17.401841 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
I20260811 11:16:17.411098 4857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:16:17.411237 4857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:16:17.412030 4857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 631a0d95ba624836a69709ab8f794a79: Sending DeleteTablet for 3 replicas of tablet 622b5d392fa0419abeb6d9cc70ad152c
I20260811 11:16:17.412328 5017 tablet_service.cc:1514] Processing DeleteTablet for tablet 622b5d392fa0419abeb6d9cc70ad152c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:17 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:54618
I20260811 11:16:17.412401 5330 tablet_service.cc:1514] Processing DeleteTablet for tablet 622b5d392fa0419abeb6d9cc70ad152c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:17 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:59292
I20260811 11:16:17.412408 5174 tablet_service.cc:1514] Processing DeleteTablet for tablet 622b5d392fa0419abeb6d9cc70ad152c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:17 UTC) from {username='oryx', principal='oryx/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:39820
I20260811 11:16:17.412495 5461 tablet_replica.cc:330] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc: stopping tablet replica
I20260811 11:16:17.412513 5463 tablet_replica.cc:330] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8: stopping tablet replica
I20260811 11:16:17.412550 5462 tablet_replica.cc:330] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: stopping tablet replica
I20260811 11:16:17.412549 5461 raft_consensus.cc:2229] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:16:17.412593 5461 raft_consensus.cc:2258] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:17.412590 5463 raft_consensus.cc:2229] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:16:17.412601 5462 raft_consensus.cc:2229] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:16:17.412637 5463 raft_consensus.cc:2258] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:17.412683 5462 raft_consensus.cc:2258] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:17.413038 5462 ts_tablet_manager.cc:1905] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:17.413036 5461 ts_tablet_manager.cc:1905] T 622b5d392fa0419abeb6d9cc70ad152c P 980e7592509d4f6ba4608a6f60076edc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:17.413056 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 4939
I20260811 11:16:17.413385 5463 ts_tablet_manager.cc:1905] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:17.415104 5462 ts_tablet_manager.cc:1918] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:16:17.415190 5462 log.cc:1196] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/622b5d392fa0419abeb6d9cc70ad152c
I20260811 11:16:17.415431 5462 ts_tablet_manager.cc:1939] T 622b5d392fa0419abeb6d9cc70ad152c P 4b419b6e7f7f4a6eb9a7853d6a21a7d2: Deleting consensus metadata
I20260811 11:16:17.415753 5463 ts_tablet_manager.cc:1918] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:16:17.415839 5463 log.cc:1196] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/622b5d392fa0419abeb6d9cc70ad152c
I20260811 11:16:17.415822 4840 catalog_manager.cc:4841] TS 4b419b6e7f7f4a6eb9a7853d6a21a7d2 (127.28.25.2:44577): tablet 622b5d392fa0419abeb6d9cc70ad152c (table test-table [id=e8bf748f6af44cd9b8a83e9f9283ea36]) successfully deleted
I20260811 11:16:17.416079 5463 ts_tablet_manager.cc:1939] T 622b5d392fa0419abeb6d9cc70ad152c P 718226ccbc2742f3a8429238ae014ae8: Deleting consensus metadata
I20260811 11:16:17.416859 4837 catalog_manager.cc:4841] TS 718226ccbc2742f3a8429238ae014ae8 (127.28.25.3:37919): tablet 622b5d392fa0419abeb6d9cc70ad152c (table test-table [id=e8bf748f6af44cd9b8a83e9f9283ea36]) successfully deleted
W20260811 11:16:17.424492 4838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:38073 (error 108)
I20260811 11:16:17.424844 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 5102
I20260811 11:16:17.428500 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:17.425047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:38073 (local address 127.0.0.1:54634)
0811 11:16:17.428354 (+ 3307us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:17.428447 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3254,"thread_start_us":47,"threads_started":1}
W20260811 11:16:17.428664 4838 catalog_manager.cc:4584] TS 980e7592509d4f6ba4608a6f60076edc (127.28.25.1:38073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 622b5d392fa0419abeb6d9cc70ad152c: Network error: Client connection negotiation failed: client connection to 127.28.25.1:38073: connect: Connection refused (error 111)
I20260811 11:16:17.434445 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 5257
I20260811 11:16:17.442430 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 4821
2026-08-11T11:16:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3092 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:16:17 dist-test-slave-2w3w krb5kdc[5541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:16:17 dist-test-slave-2w3w krb5kdc[5541](info): set up 2 sockets
Aug 11 11:16:17 dist-test-slave-2w3w krb5kdc[5541](info): commencing operation
krb5kdc: starting...
W20260811 11:16:18.591199 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:16:18 dist-test-slave-2w3w krb5kdc[5541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:16:18.735474 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39339
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:39339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:18.827085 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:18.827354 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:18.827382 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:18.828733 5557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:16:18.828801 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:18.828816 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:18.828830 5557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:16:18.828842 5557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:16:18.830575 5557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:39339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:18.830904 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:18.831204 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:18.837014 5563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:18.837007 5562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:18.837262 5557 server_base.cc:1034] running on GCE node
W20260811 11:16:18.837006 5566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:18.837756 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:18.838150 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:18.839437 5557 hybrid_clock.cc:648] HybridClock initialized: now 1786446978839409 us; error 47 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-11T11:16:18Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1398 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:16:18 dist-test-slave-2w3w krb5kdc[5572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:16:18 dist-test-slave-2w3w krb5kdc[5572](info): set up 2 sockets
Aug 11 11:16:18 dist-test-slave-2w3w krb5kdc[5572](info): commencing operation
krb5kdc: starting...
W20260811 11:16:19.970943 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:16:20 dist-test-slave-2w3w krb5kdc[5572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:16:20.120210 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:37543
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:37543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:20.207366 5586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:20.207679 5586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:20.207726 5586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:20.209101 5586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:16:20.209204 5586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:20.209230 5586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:20.209252 5586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:16:20.209285 5586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:16:20.211457 5586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:37543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:37543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:20.211974 5586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:20.212322 5586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:20.218245 5594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:20.218250 5592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:20.218279 5591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:20.218654 5586 server_base.cc:1034] running on GCE node
I20260811 11:16:20.219129 5586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:20.219511 5586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:20.220722 5586 hybrid_clock.cc:648] HybridClock initialized: now 1786446980220677 us; error 66 us; skew 500 ppm
Aug 11 11:16:20 dist-test-slave-2w3w krb5kdc[5572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446980, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:20.230381 5586 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:16:20.230890 5586 webserver.cc:460] Webserver started at http://127.28.25.62:36993/ using document root <none> and password file <none>
I20260811 11:16:20.231220 5586 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:20.231276 5586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:20.231422 5586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:20.232637 5586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "181b98e9d834475e92f7d9f577e43dc8"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "ab7092c11e641cac601f3404566b5f37"
server_key_iv: "9115847e5507417d6fb2ed60543cff66"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.233076 5586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "181b98e9d834475e92f7d9f577e43dc8"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "ab7092c11e641cac601f3404566b5f37"
server_key_iv: "9115847e5507417d6fb2ed60543cff66"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.234726 5586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:16:20.235858 5601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:20.236202 5586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:20.236364 5586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "181b98e9d834475e92f7d9f577e43dc8"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "ab7092c11e641cac601f3404566b5f37"
server_key_iv: "9115847e5507417d6fb2ed60543cff66"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.236482 5586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:20.255152 5586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:20.257992 5586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:20.258215 5586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:20.264079 5586 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:37543
I20260811 11:16:20.264180 5659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:37543 every 8 connection(s)
I20260811 11:16:20.264602 5586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:16:20.265663 5660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:20.267263 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 5586
I20260811 11:16:20.267413 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:16:20.267643 28772 external_mini_cluster.cc:1428] Setting key 815ab8eb344e36864a351e2e7c41751d
I20260811 11:16:20.269354 5660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8: Bootstrap starting.
I20260811 11:16:20.270308 5660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:20.270673 5660 log.cc:824] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:20.271631 5660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8: No bootstrap required, opened a new log
I20260811 11:16:20.274484 5660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } }
I20260811 11:16:20.274762 5660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:20.274817 5660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b98e9d834475e92f7d9f577e43dc8, State: Initialized, Role: FOLLOWER
I20260811 11:16:20.275034 5660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } }
I20260811 11:16:20.275159 5660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:16:20.275206 5660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:16:20.275261 5660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 0 FOLLOWER]: Advancing to term 1
Aug 11 11:16:20 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:16:20.276037 5660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } }
I20260811 11:16:20.276178 5660 leader_election.cc:302] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181b98e9d834475e92f7d9f577e43dc8; no voters:
I20260811 11:16:20.276440 5660 leader_election.cc:288] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:16:20.276472 5667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:20.276686 5667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 181b98e9d834475e92f7d9f577e43dc8, State: Running, Role: LEADER
I20260811 11:16:20.276790 5667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } }
I20260811 11:16:20.276882 5660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:16:20.277189 5667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "181b98e9d834475e92f7d9f577e43dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } } }
I20260811 11:16:20.277357 5667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:20.277609 5667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181b98e9d834475e92f7d9f577e43dc8. Latest consensus state: current_term: 1 leader_uuid: "181b98e9d834475e92f7d9f577e43dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } } }
I20260811 11:16:20.277701 5667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:20.278025 5673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:16:20.278620 5673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:16:20.281087 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:20.268315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.28.25.62:37543)
0811 11:16:20.268559 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:20.268568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:20.268597 (+ 29us) server_negotiation.cc:407] Connection header received
0811 11:16:20.269044 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:20.269050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:20.269103 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:20.269252 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:20.270226 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.271481 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.272667 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.273146 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.273246 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:20.276537 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:20.276547 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:20.276547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:20.276561 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:20.279188 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:20.279658 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:20.279662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:20.279663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:20.279791 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:20.280061 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:20.280062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:20.280063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:20.280430 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:20.280606 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:20.280685 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 11:16:20.280848 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":120,"threads_started":1}
I20260811 11:16:20.283844 5673 catalog_manager.cc:1347] Generated new cluster ID: 6409de45885f4e719750601eec4c79ac
I20260811 11:16:20.283927 5673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:16:20.323112 5673 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:16:20.323740 5673 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:16:20.345201 5673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8: Generated new TSK 0
I20260811 11:16:20.345485 5673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:16:20.395824 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37543
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:20.492676 5692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:20.492924 5692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:20.492955 5692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:20.494212 5692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:20.494261 5692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:20.494290 5692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:16:20.495975 5692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:20.496222 5692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:20.496419 5692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:20.501791 5697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:20.501778 5700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:20.501912 5692 server_base.cc:1034] running on GCE node
W20260811 11:16:20.501776 5698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:20.502326 5692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:20.502617 5692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:20.503878 5692 hybrid_clock.cc:648] HybridClock initialized: now 1786446980503862 us; error 49 us; skew 500 ppm
Aug 11 11:16:20 dist-test-slave-2w3w krb5kdc[5572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446980, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:20.506636 5692 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:16:20.507165 5692 webserver.cc:460] Webserver started at http://127.28.25.1:33525/ using document root <none> and password file <none>
I20260811 11:16:20.507395 5692 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:20.507441 5692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:20.507540 5692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:20.508625 5692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "a400d93d6c0f4ff0b5170530753d613f"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "72e5af9ee281615970d4d6d1a958c17c"
server_key_iv: "760b29a1b790245bdb26905f928de6b7"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.508973 5692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "a400d93d6c0f4ff0b5170530753d613f"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "72e5af9ee281615970d4d6d1a958c17c"
server_key_iv: "760b29a1b790245bdb26905f928de6b7"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.510596 5692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:16:20.511575 5709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:20.511904 5692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:16:20.512002 5692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "a400d93d6c0f4ff0b5170530753d613f"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "72e5af9ee281615970d4d6d1a958c17c"
server_key_iv: "760b29a1b790245bdb26905f928de6b7"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.512068 5692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:20.524961 5692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:20.527583 5692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:20.527719 5692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:20.527922 5692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:20.528286 5692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:20.528332 5692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:20.528357 5692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:20.528369 5692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:20.535286 5692 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:44375
I20260811 11:16:20.535332 5839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:44375 every 8 connection(s)
I20260811 11:16:20.535713 5692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
Aug 11 11:16:20 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446980, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:16:20.543293 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 5692
I20260811 11:16:20.543452 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:16:20.543721 28772 external_mini_cluster.cc:1428] Setting key 58cf85b4c8ab4b735afefcfb8372eb56
I20260811 11:16:20.547499 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:20.536622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:45879 (local address 127.28.25.62:37543)
0811 11:16:20.536736 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:20.536738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:20.536936 (+ 198us) server_negotiation.cc:407] Connection header received
0811 11:16:20.537561 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:20.537563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:20.537576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:20.537639 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:20.538831 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.539461 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.540677 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.540991 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.541075 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:20.543711 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:20.543712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:20.543712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:20.543721 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:20.545653 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:20.546184 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:20.546186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:20.546186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:20.546259 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:20.546584 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:20.546586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:20.546586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:20.546729 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:20.546863 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:20.547250 (+ 387us) server_negotiation.cc:299] Negotiation successful
0811 11:16:20.547398 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 11:16:20.547668 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:20.536517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:45879)
0811 11:16:20.536781 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:20.536831 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:20.537431 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:20.537718 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:20.537728 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:20.538448 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:20.538680 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:20.538686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.539595 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.539596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:20.540565 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:20.540568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.541091 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.541092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:20.541237 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:20.541868 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:20.541878 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:20.543574 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:20.545776 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:20.545786 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:20.545789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:20.546091 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:20.546347 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:20.546347 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:20.546348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:20.546472 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:20.546851 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:20.546860 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:20.547091 (+ 231us) client_negotiation.cc:769] Sending connection context
0811 11:16:20.547216 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 11:16:20.547410 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":97,"threads_started":1}
I20260811 11:16:20.548254 5840 heartbeater.cc:347] Connected to a master server at 127.28.25.62:37543
I20260811 11:16:20.548382 5840 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:20.548604 5840 heartbeater.cc:511] Master 127.28.25.62:37543 requested a full tablet report, sending...
I20260811 11:16:20.549213 5619 ts_manager.cc:194] Registered new tserver with Master: a400d93d6c0f4ff0b5170530753d613f (127.28.25.1:44375)
I20260811 11:16:20.549894 5619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:45879
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:16:20.579814 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37543
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:20.672331 5847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:20.672644 5847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:20.672694 5847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:20.674034 5847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:20.674137 5847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:20.674190 5847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:16:20.676025 5847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:20.676452 5847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:20.676738 5847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:20.682842 5855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:20.682878 5852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:20.682842 5853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:20.683079 5847 server_base.cc:1034] running on GCE node
I20260811 11:16:20.683419 5847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:20.683706 5847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:20.684872 5847 hybrid_clock.cc:648] HybridClock initialized: now 1786446980684823 us; error 60 us; skew 500 ppm
Aug 11 11:16:20 dist-test-slave-2w3w krb5kdc[5572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446980, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:20.687803 5847 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:16:20.688303 5847 webserver.cc:460] Webserver started at http://127.28.25.2:40419/ using document root <none> and password file <none>
I20260811 11:16:20.688586 5847 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:20.688664 5847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:20.688827 5847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:20.689993 5847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "224e958e62434c1d89feea3a7a3c4c77"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "4640c5508e25037ff4222d0e6c7bce0e"
server_key_iv: "7ffd58a6b3613296775eb5f249a33b7c"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.690411 5847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "224e958e62434c1d89feea3a7a3c4c77"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "4640c5508e25037ff4222d0e6c7bce0e"
server_key_iv: "7ffd58a6b3613296775eb5f249a33b7c"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.692052 5847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:16:20.693015 5863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:20.693266 5847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:20.693403 5847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "224e958e62434c1d89feea3a7a3c4c77"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "4640c5508e25037ff4222d0e6c7bce0e"
server_key_iv: "7ffd58a6b3613296775eb5f249a33b7c"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.693495 5847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:20.714690 5847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:20.717459 5847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:20.717629 5847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:20.717927 5847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:20.718288 5847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:20.718325 5847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:20.718349 5847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:20.718398 5847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:20.724866 5847 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:37247
I20260811 11:16:20.724965 5991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:37247 every 8 connection(s)
I20260811 11:16:20.725339 5847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:16:20.726692 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 5847
I20260811 11:16:20.726825 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:16:20.727113 28772 external_mini_cluster.cc:1428] Setting key 6c6aef7aa40f2955de0807244651e424
Aug 11 11:16:20 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446980, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
I20260811 11:16:20.737974 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:20.726324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:40537 (local address 127.28.25.62:37543)
0811 11:16:20.726440 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:20.726459 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:20.726629 (+ 170us) server_negotiation.cc:407] Connection header received
0811 11:16:20.728009 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:20.728011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:20.728023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:20.728094 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:20.728774 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.729343 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.730929 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.731230 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.731308 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:20.734508 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:20.734509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:20.734510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:20.734518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:20.736286 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:20.736859 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:20.736861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:20.736861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:20.736928 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:20.737211 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:20.737212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:20.737213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:20.737392 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:20.737471 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:20.737781 (+ 310us) server_negotiation.cc:299] Negotiation successful
0811 11:16:20.737877 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 11:16:20.738305 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:20.726280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:40537)
0811 11:16:20.726470 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:20.726519 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:20.727862 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:20.728160 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:20.728163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:20.728445 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:20.728662 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:20.728666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.729476 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.729477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:20.730753 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:20.730758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.731345 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.731346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:20.731506 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:20.732149 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:20.732162 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:20.734338 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:20.736444 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:20.736456 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:20.736459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:20.736759 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:20.737003 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:20.737004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:20.737005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:20.737128 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:20.737460 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:20.737470 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:20.737627 (+ 157us) client_negotiation.cc:769] Sending connection context
0811 11:16:20.737744 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 11:16:20.738029 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
I20260811 11:16:20.738997 5992 heartbeater.cc:347] Connected to a master server at 127.28.25.62:37543
I20260811 11:16:20.739138 5992 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:20.739349 5992 heartbeater.cc:511] Master 127.28.25.62:37543 requested a full tablet report, sending...
I20260811 11:16:20.739938 5619 ts_manager.cc:194] Registered new tserver with Master: 224e958e62434c1d89feea3a7a3c4c77 (127.28.25.2:37247)
I20260811 11:16:20.740345 5619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:40537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:16:20.764250 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37543
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:20.854130 5998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:20.854430 5998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:20.854480 5998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:20.855880 5998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:20.855988 5998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:20.856035 5998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:16:20.857687 5998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:20.858018 5998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:20.858451 5998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:20.865324 6005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:20.865309 6004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:20.865304 6007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:20.865645 5998 server_base.cc:1034] running on GCE node
I20260811 11:16:20.865948 5998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:20.866338 5998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:20.867565 5998 hybrid_clock.cc:648] HybridClock initialized: now 1786446980867524 us; error 62 us; skew 500 ppm
Aug 11 11:16:20 dist-test-slave-2w3w krb5kdc[5572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446980, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:20.870474 5998 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:16:20.871063 5998 webserver.cc:460] Webserver started at http://127.28.25.3:38773/ using document root <none> and password file <none>
I20260811 11:16:20.871272 5998 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:20.871315 5998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:20.871416 5998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:20.872517 5998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "65d58148940a4aaa8a997e810b13493b"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "f07b4435fad71f54355a1f587280fe2b"
server_key_iv: "b9583499b51888d195ae2a9b371f2c9d"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.872839 5998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "65d58148940a4aaa8a997e810b13493b"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "f07b4435fad71f54355a1f587280fe2b"
server_key_iv: "b9583499b51888d195ae2a9b371f2c9d"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.874238 5998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:16:20.875228 6015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:20.875411 5998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:16:20.875524 5998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "65d58148940a4aaa8a997e810b13493b"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "f07b4435fad71f54355a1f587280fe2b"
server_key_iv: "b9583499b51888d195ae2a9b371f2c9d"
server_key_version: "encryptionkey@0"
I20260811 11:16:20.875586 5998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:20.890089 5998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:20.892756 5998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:20.892887 5998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:20.893134 5998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:20.893507 5998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:20.893546 5998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:20.893571 5998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:20.893584 5998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:20.900372 5998 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:37073
I20260811 11:16:20.900487 6141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:37073 every 8 connection(s)
I20260811 11:16:20.900832 5998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:16:20.901487 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 5998
I20260811 11:16:20.901599 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:16:20.901829 28772 external_mini_cluster.cc:1428] Setting key da516e1fd0fd357e1f70357258aad401
Aug 11 11:16:20 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446980, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:16:20.912586 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:20.901772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:55111)
0811 11:16:20.901993 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:20.902048 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:20.902791 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:20.903079 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:20.903083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:20.903723 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:20.903945 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:20.903951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.904708 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.904709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:20.905618 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:20.905621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.906183 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.906184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:20.906322 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:20.907086 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:20.907096 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:20.908797 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:20.910662 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:20.910672 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:20.910674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:20.911100 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:20.911431 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:20.911432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:20.911432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:20.911579 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:20.911932 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:20.911940 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:20.912093 (+ 153us) client_negotiation.cc:769] Sending connection context
0811 11:16:20.912187 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 11:16:20.912373 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
I20260811 11:16:20.912623 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:20.902147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:55111 (local address 127.28.25.62:37543)
0811 11:16:20.902195 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:20.902196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:20.902205 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:16:20.902902 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:20.902902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:20.902916 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:20.903013 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:20.904066 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.904592 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.905785 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:20.906027 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:20.906107 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:20.908933 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:20.908935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:20.908935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:20.908945 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:20.910547 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:20.911250 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:20.911252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:20.911252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:20.911320 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:20.911695 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:20.911696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:20.911696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:20.911856 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:20.911918 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:20.912293 (+ 375us) server_negotiation.cc:299] Negotiation successful
0811 11:16:20.912553 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:16:20.913262 6142 heartbeater.cc:347] Connected to a master server at 127.28.25.62:37543
I20260811 11:16:20.913391 6142 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:20.913784 6142 heartbeater.cc:511] Master 127.28.25.62:37543 requested a full tablet report, sending...
I20260811 11:16:20.914305 5619 ts_manager.cc:194] Registered new tserver with Master: 65d58148940a4aaa8a997e810b13493b (127.28.25.3:37073)
I20260811 11:16:20.914772 5619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:55111
I20260811 11:16:20.925459 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab.
Entry for principal oryx/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab.
WARNING: no policy specified for oryx/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab.
Entry for principal oryx/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab.
WARNING: no policy specified for oryx/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab.
Entry for principal oryx/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab.
I20260811 11:16:21.022600 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 5692
I20260811 11:16:21.030342 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 5847
I20260811 11:16:21.038280 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 5998
I20260811 11:16:21.044973 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 5586
I20260811 11:16:21.054625 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:37543
--webserver_interface=127.28.25.62
--webserver_port=36993
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:37543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:21.149783 6154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:21.150072 6154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:21.150102 6154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:21.151607 6154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:16:21.151674 6154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:21.151690 6154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:21.151705 6154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:16:21.151718 6154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:16:21.153519 6154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:37543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:37543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=36993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:21.153784 6154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:21.154021 6154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:21.159832 6164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:21.159932 6162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:21.159864 6161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:21.160180 6154 server_base.cc:1034] running on GCE node
I20260811 11:16:21.160562 6154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:21.160935 6154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:21.162110 6154 hybrid_clock.cc:648] HybridClock initialized: now 1786446981162083 us; error 40 us; skew 500 ppm
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446981, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:21.165406 6154 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:16:21.166024 6154 webserver.cc:460] Webserver started at http://127.28.25.62:36993/ using document root <none> and password file <none>
I20260811 11:16:21.166285 6154 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:21.166416 6154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:21.168630 6154 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:21.169499 6171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:21.169703 6154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:21.169827 6154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "181b98e9d834475e92f7d9f577e43dc8"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "ab7092c11e641cac601f3404566b5f37"
server_key_iv: "9115847e5507417d6fb2ed60543cff66"
server_key_version: "encryptionkey@0"
I20260811 11:16:21.170264 6154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:21.195127 6154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:21.197992 6154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:21.198189 6154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:21.202997 6154 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:37543
I20260811 11:16:21.203083 6227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:37543 every 8 connection(s)
I20260811 11:16:21.203679 6154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:16:21.205484 6228 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:16:21.206120 6228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8: Bootstrap starting.
I20260811 11:16:21.211648 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 6154
I20260811 11:16:21.212041 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:44375
--local_ip_for_outbound_sockets=127.28.25.1
--tserver_master_addrs=127.28.25.62:37543
--webserver_port=33525
--webserver_interface=127.28.25.1
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.25.1
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:16:21.216279 6228 log.cc:824] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:21.218142 6228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:16:21.218501 6228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8: Bootstrap complete.
I20260811 11:16:21.221103 6228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } }
I20260811 11:16:21.221376 6228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b98e9d834475e92f7d9f577e43dc8, State: Initialized, Role: FOLLOWER
I20260811 11:16:21.221555 6228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } }
I20260811 11:16:21.221624 6228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:16:21.221670 6228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:16:21.221725 6228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 1 FOLLOWER]: Advancing to term 2
I20260811 11:16:21.222905 6228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } }
I20260811 11:16:21.223088 6228 leader_election.cc:302] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181b98e9d834475e92f7d9f577e43dc8; no voters:
I20260811 11:16:21.223376 6228 leader_election.cc:288] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 11:16:21.223417 6232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 11:16:21.223541 6232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [term 2 LEADER]: Becoming Leader. State: Replica: 181b98e9d834475e92f7d9f577e43dc8, State: Running, Role: LEADER
I20260811 11:16:21.223690 6228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:16:21.223819 6232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } }
I20260811 11:16:21.224141 6233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "181b98e9d834475e92f7d9f577e43dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } } }
I20260811 11:16:21.224254 6233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:21.224129 6234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181b98e9d834475e92f7d9f577e43dc8. Latest consensus state: current_term: 2 leader_uuid: "181b98e9d834475e92f7d9f577e43dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181b98e9d834475e92f7d9f577e43dc8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37543 } } }
I20260811 11:16:21.224574 6234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:21.224822 6240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:16:21.225422 6240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:16:21.225559 6240 catalog_manager.cc:1259] Loaded cluster ID: 6409de45885f4e719750601eec4c79ac
I20260811 11:16:21.225598 6240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:16:21.226773 6240 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:16:21.227010 6240 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 181b98e9d834475e92f7d9f577e43dc8: Loaded TSK: 0
I20260811 11:16:21.227245 6240 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 11:16:21.304672 6230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:21.304966 6230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:21.305015 6230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:21.306293 6230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:21.306385 6230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:21.306433 6230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:16:21.308660 6230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.1.keytab
--principal=oryx/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:44375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=33525
--tserver_master_addrs=127.28.25.62:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:21.309034 6230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:21.309337 6230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:21.315956 6257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:21.316159 6259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:21.315943 6256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:21.316506 6230 server_base.cc:1034] running on GCE node
I20260811 11:16:21.316817 6230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:21.317127 6230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:21.318387 6230 hybrid_clock.cc:648] HybridClock initialized: now 1786446981318334 us; error 66 us; skew 500 ppm
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446981, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:21.321652 6230 init.cc:377] Logged in from keytab as oryx/127.28.25.1@KRBTEST.COM (short username oryx)
I20260811 11:16:21.322402 6230 webserver.cc:460] Webserver started at http://127.28.25.1:33525/ using document root <none> and password file <none>
I20260811 11:16:21.322716 6230 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:21.322826 6230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:21.324855 6230 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:21.325830 6271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:21.326084 6230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:21.326201 6230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "a400d93d6c0f4ff0b5170530753d613f"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "72e5af9ee281615970d4d6d1a958c17c"
server_key_iv: "760b29a1b790245bdb26905f928de6b7"
server_key_version: "encryptionkey@0"
I20260811 11:16:21.326606 6230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:21.352283 6230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:21.354931 6230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:21.355155 6230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:21.355428 6230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:21.355841 6230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:21.355906 6230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:21.355958 6230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:21.355988 6230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:21.362514 6230 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:44375
I20260811 11:16:21.362599 6400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:44375 every 8 connection(s)
I20260811 11:16:21.363045 6230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:16:21.369349 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 6230
I20260811 11:16:21.369689 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:37247
--local_ip_for_outbound_sockets=127.28.25.2
--tserver_master_addrs=127.28.25.62:37543
--webserver_port=40419
--webserver_interface=127.28.25.2
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.25.2
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:21.373631 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:21.363798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:53947)
0811 11:16:21.364025 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:21.364104 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:21.364696 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:21.365250 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:21.365255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:21.365938 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:21.366159 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.366165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.367592 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.367593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.368660 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.368664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.369350 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.369351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.369509 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.370215 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:21.370226 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:21.373394 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":116,"threads_started":1}
W20260811 11:16:21.373749 6404 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:21.363957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:53947 (local address 127.28.25.62:37543)
0811 11:16:21.364194 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:21.364203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:21.364250 (+ 47us) server_negotiation.cc:407] Connection header received
0811 11:16:21.364950 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:21.364958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:21.365025 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:21.365183 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:21.366355 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.367454 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.368764 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.369237 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.369385 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.373571 (+ 4186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:53947: BlockingRecv error: recv got EOF from 127.28.25.1:53947 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
W20260811 11:16:21.373919 6401 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:37543 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.25.62:37543: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:21.380193 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:21.374239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:46583)
0811 11:16:21.374403 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:21.374418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:21.374490 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:21.374661 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:21.374663 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:21.374880 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:21.374986 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.374989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.375689 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.375690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.376736 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.376740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.377174 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.377175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.377266 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.377778 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:21.377786 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:21.380102 (+ 2316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137}
W20260811 11:16:21.380276 6404 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:21.374325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:46583 (local address 127.28.25.62:37543)
0811 11:16:21.374423 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:21.374425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:21.374468 (+ 43us) server_negotiation.cc:407] Connection header received
0811 11:16:21.374555 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:21.374556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:21.374569 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:21.374619 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:21.375083 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.375596 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.376857 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.377080 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.377147 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.380208 (+ 3061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:46583: BlockingRecv error: recv got EOF from 127.28.25.1:46583 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:21.384975 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:21.380564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:39869)
0811 11:16:21.380613 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:21.380624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:21.380667 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:21.380798 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:21.380799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:21.380950 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:21.380989 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.380990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.381649 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.381650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.382153 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.382154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.382550 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.382551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.382611 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.383109 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:21.383115 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:21.384889 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260811 11:16:21.385110 6404 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:21.380617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:39869 (local address 127.28.25.62:37543)
0811 11:16:21.380658 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:21.380660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:21.380669 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:16:21.380733 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:21.380734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:21.380745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:21.380795 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:21.381073 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.381560 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.382257 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.382459 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.382549 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.385022 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:39869: BlockingRecv error: recv got EOF from 127.28.25.1:39869 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260811 11:16:21.385201 6401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:16:21.463178 6405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:21.463445 6405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:21.463490 6405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:21.464859 6405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:21.464958 6405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:21.465008 6405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:16:21.466619 6405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.2.keytab
--principal=oryx/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:37247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=40419
--tserver_master_addrs=127.28.25.62:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:21.466946 6405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:21.467278 6405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:21.473121 6410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:21.473105 6414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:21.473097 6412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:21.473142 6405 server_base.cc:1034] running on GCE node
I20260811 11:16:21.473675 6405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:21.473995 6405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:21.475169 6405 hybrid_clock.cc:648] HybridClock initialized: now 1786446981475149 us; error 43 us; skew 500 ppm
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446981, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:21.478070 6405 init.cc:377] Logged in from keytab as oryx/127.28.25.2@KRBTEST.COM (short username oryx)
I20260811 11:16:21.478700 6405 webserver.cc:460] Webserver started at http://127.28.25.2:40419/ using document root <none> and password file <none>
I20260811 11:16:21.479072 6405 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:21.479173 6405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:21.480857 6405 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:21.481688 6421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:21.481937 6405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:21.482051 6405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "224e958e62434c1d89feea3a7a3c4c77"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "4640c5508e25037ff4222d0e6c7bce0e"
server_key_iv: "7ffd58a6b3613296775eb5f249a33b7c"
server_key_version: "encryptionkey@0"
I20260811 11:16:21.482462 6405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:21.500867 6405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:21.503377 6405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:21.503557 6405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:21.503820 6405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:21.504151 6405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:21.504211 6405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:21.504252 6405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:21.504280 6405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:21.511050 6405 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:37247
I20260811 11:16:21.511502 6551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:37247 every 8 connection(s)
I20260811 11:16:21.511610 6405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:16:21.517402 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 6405
I20260811 11:16:21.517755 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:37073
--local_ip_for_outbound_sockets=127.28.25.3
--tserver_master_addrs=127.28.25.62:37543
--webserver_port=38773
--webserver_interface=127.28.25.3
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.25.3
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:21.520064 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:21.512450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:55829)
0811 11:16:21.512641 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:21.512681 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:21.513237 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:21.513488 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:21.513492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:21.514081 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:21.514288 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.514294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.515156 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.515158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.515995 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.515997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.516374 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.516375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.516491 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.517112 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:21.517122 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:21.519893 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
W20260811 11:16:21.520114 6404 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:21.512451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:55829 (local address 127.28.25.62:37543)
0811 11:16:21.512520 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:21.512522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:21.512768 (+ 246us) server_negotiation.cc:407] Connection header received
0811 11:16:21.513353 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:21.513355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:21.513369 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:21.513423 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:21.514402 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.515057 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.516075 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.516266 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.516324 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.520031 (+ 3707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:55829: BlockingRecv error: recv got EOF from 127.28.25.2:55829 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260811 11:16:21.520382 6552 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:37543 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.25.62:37543: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:21.525537 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:21.520669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:36363)
0811 11:16:21.520747 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:21.520760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:21.520835 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:21.521024 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:21.521025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:21.521169 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:21.521224 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.521226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.521903 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.521904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.522494 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.522497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.522921 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.522923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.523028 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.523520 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:21.523525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:21.525439 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 11:16:21.525591 6404 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:21.520784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:36363 (local address 127.28.25.62:37543)
0811 11:16:21.520862 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:21.520864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:21.520872 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:16:21.520948 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:21.520949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:21.520960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:21.521018 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:21.521309 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.521809 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.522585 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.522842 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.522923 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.525524 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:36363: BlockingRecv error: recv got EOF from 127.28.25.2:36363 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:21.530174 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:21.525823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:54917)
0811 11:16:21.526008 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:21.526017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:21.526083 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:21.526207 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:21.526208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:21.526341 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:21.526375 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.526376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.527055 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.527056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.527554 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.527556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.527918 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.527919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.527972 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.528428 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:21.528433 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:21.530080 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":124,"threads_started":1}
W20260811 11:16:21.530226 6404 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:21.525870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:54917 (local address 127.28.25.62:37543)
0811 11:16:21.525908 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:21.525910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:21.526070 (+ 160us) server_negotiation.cc:407] Connection header received
0811 11:16:21.526135 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:21.526136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:21.526146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:21.526202 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:21.526459 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.526934 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.527642 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.527842 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.527921 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.530159 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:54917: BlockingRecv error: recv got EOF from 127.28.25.2:54917 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260811 11:16:21.530323 6552 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:16:21.607115 6555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:21.607385 6555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:21.607432 6555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:21.608722 6555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:21.608819 6555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:21.608870 6555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:16:21.610445 6555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/krb5kdc/oryx_127.28.25.3.keytab
--principal=oryx/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:37073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=38773
--tserver_master_addrs=127.28.25.62:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:21.610772 6555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:21.611068 6555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:21.616906 6561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:21.616906 6562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:21.616906 6564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:21.617244 6555 server_base.cc:1034] running on GCE node
I20260811 11:16:21.617424 6555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:21.617643 6555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:21.618810 6555 hybrid_clock.cc:648] HybridClock initialized: now 1786446981618796 us; error 31 us; skew 500 ppm
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446981, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:21.621841 6555 init.cc:377] Logged in from keytab as oryx/127.28.25.3@KRBTEST.COM (short username oryx)
I20260811 11:16:21.622399 6555 webserver.cc:460] Webserver started at http://127.28.25.3:38773/ using document root <none> and password file <none>
I20260811 11:16:21.622655 6555 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:21.622747 6555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:21.624572 6555 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:21.625478 6571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:21.625669 6555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:21.625773 6555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "65d58148940a4aaa8a997e810b13493b"
format_stamp: "Formatted at 2026-08-11 11:16:20 on dist-test-slave-2w3w"
server_key: "f07b4435fad71f54355a1f587280fe2b"
server_key_iv: "b9583499b51888d195ae2a9b371f2c9d"
server_key_version: "encryptionkey@0"
I20260811 11:16:21.626276 6555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:21.643311 6555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:21.645776 6555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:21.645903 6555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:21.646119 6555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:21.646539 6555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:21.646577 6555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:21.646601 6555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:21.646651 6555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:21.653088 6555 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:37073
I20260811 11:16:21.653192 6697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:37073 every 8 connection(s)
I20260811 11:16:21.653563 6555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:16:21.654510 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 6555
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 11:16:21.662372 6404 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:21.654458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:57447 (local address 127.28.25.62:37543)
0811 11:16:21.654536 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:21.654539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:21.654822 (+ 283us) server_negotiation.cc:407] Connection header received
0811 11:16:21.655646 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:21.655647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:21.655659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:21.655757 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:21.657200 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.657687 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.658749 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.659025 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.659087 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.662301 (+ 3214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:57447: BlockingRecv error: recv got EOF from 127.28.25.3:57447 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:16:21.662489 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:21.654430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:57447)
0811 11:16:21.654640 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:21.654685 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:21.655514 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:21.655905 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:21.655910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:21.656750 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:21.657101 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.657107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.657810 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.657811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.658649 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.658652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.659099 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.659100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.659236 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.659828 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:21.659836 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:21.662227 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":130,"threads_started":1}
W20260811 11:16:21.662799 6698 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:37543 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.25.62:37543: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
I20260811 11:16:21.664534 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:21.655201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.28.25.62:37543)
0811 11:16:21.655794 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:21.655796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:21.655822 (+ 26us) server_negotiation.cc:407] Connection header received
0811 11:16:21.655845 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:21.655846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:21.655857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:21.655931 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:21.656990 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.657569 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.658541 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.658786 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.658875 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.660510 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:21.660515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:21.660516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:21.660537 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:21.662585 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:21.663438 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:21.663441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:21.663442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:21.663554 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:21.663771 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:21.663772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:21.663772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:21.664024 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:21.664195 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:21.664306 (+ 111us) server_negotiation.cc:299] Negotiation successful
0811 11:16:21.664424 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":61,"threads_started":1}
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:21.667883 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:21.663145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:41279)
0811 11:16:21.663181 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:21.663192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:21.663254 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:21.663366 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:21.663367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:21.663501 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:21.663564 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.663565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.664211 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.664212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.664939 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.664943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.665396 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.665397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.665474 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.665811 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:21.665815 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:21.667830 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260811 11:16:21.668023 6404 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:21.663200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:41279 (local address 127.28.25.62:37543)
0811 11:16:21.663262 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:21.663264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:21.663275 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:16:21.663307 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:21.663308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:21.663318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:21.663371 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:21.663650 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.664128 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.665038 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.665319 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.665395 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.667935 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:41279: BlockingRecv error: recv got EOF from 127.28.25.3:41279 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:21 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:21.673166 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:21.668287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:50957)
0811 11:16:21.668351 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:21.668363 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:21.668410 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:21.668571 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:21.668572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:21.668717 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:21.668752 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.668754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.669670 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.669672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.670262 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:21.670264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.670697 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.670699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:21.670766 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.671383 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:21.671387 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:21.673071 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 11:16:21.673219 6404 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:21.668361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:50957 (local address 127.28.25.62:37543)
0811 11:16:21.668408 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:21.668410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:21.668421 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:16:21.668472 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:21.668473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:21.668483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:21.668567 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:21.668862 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.669578 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.670363 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:21.670599 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:21.670683 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:21.673153 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:50957: BlockingRecv error: recv got EOF from 127.28.25.3:50957 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260811 11:16:21.673383 6698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 11 11:16:22 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:22 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:22.391546 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:22.385836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:59059)
0811 11:16:22.386042 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:22.386056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:22.386136 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:22.386407 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:22.386409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:22.386587 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:22.386642 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:22.386643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.387539 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.387541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:22.388301 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:22.388304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.388710 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.388711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:22.388767 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:22.389382 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:22.389388 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:22.391471 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":105,"threads_started":1}
W20260811 11:16:22.391711 6705 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:22.385937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:59059 (local address 127.28.25.62:37543)
0811 11:16:22.386097 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:22.386099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:22.386115 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:16:22.386199 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:22.386199 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:22.386211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:22.386365 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:22.386720 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.387433 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.388380 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.388595 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.388659 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:22.391620 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:59059: BlockingRecv error: recv got EOF from 127.28.25.1:59059 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
Aug 11 11:16:22 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:22 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:22.536605 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:22.530845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:50191)
0811 11:16:22.531027 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:22.531044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:22.531116 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:22.531244 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:22.531245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:22.531442 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:22.531497 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:22.531499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.532252 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.532253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:22.532823 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:22.532825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.533233 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.533235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:22.533291 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:22.533999 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:22.534005 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:22.536511 (+ 2506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
W20260811 11:16:22.536806 6705 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:22.530990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:50191 (local address 127.28.25.62:37543)
0811 11:16:22.531059 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:22.531061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:22.531120 (+ 59us) server_negotiation.cc:407] Connection header received
0811 11:16:22.531163 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:22.531164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:22.531177 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:22.531230 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:22.531599 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.532162 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.532923 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.533126 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.533193 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:22.536731 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:50191: BlockingRecv error: recv got EOF from 127.28.25.2:50191 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 11 11:16:22 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:22 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:22.679337 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:22.673840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:46893)
0811 11:16:22.673996 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:22.674008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:22.674090 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:22.674520 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:22.674522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:22.674723 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:22.674790 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:22.674793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.675612 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.675613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:22.676206 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:22.676208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.676606 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.676608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:22.676665 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:22.677193 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:22.677198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:22.679260 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
W20260811 11:16:22.679498 6705 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:22.673941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:46893 (local address 127.28.25.62:37543)
0811 11:16:22.674007 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:22.674009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:22.674094 (+ 85us) server_negotiation.cc:407] Connection header received
0811 11:16:22.674373 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:22.674375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:22.674390 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:22.674453 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:22.674917 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.675521 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.676297 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:22.676502 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:22.676559 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:22.679409 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:46893: BlockingRecv error: recv got EOF from 127.28.25.3:46893 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 11 11:16:23 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:23 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:23.397248 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:23.392148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:58067)
0811 11:16:23.392323 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:23.392338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:23.392425 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:23.392594 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:23.392594 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:23.392770 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:23.392825 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:23.392827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.393562 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.393563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:23.394120 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:23.394122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.394474 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.394475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:23.394516 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:23.395074 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:23.395078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:23.397149 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
W20260811 11:16:23.397356 6709 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:23.392267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:58067 (local address 127.28.25.62:37543)
0811 11:16:23.392401 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:23.392403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:23.392416 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:16:23.392481 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:23.392482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:23.392493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:23.392570 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:23.392949 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.393472 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.394202 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.394403 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.394456 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:23.397278 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:58067: BlockingRecv error: recv got EOF from 127.28.25.1:58067 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
Aug 11 11:16:23 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:23 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:23.543079 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:23.537208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:34365)
0811 11:16:23.537389 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:23.537402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:23.537477 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:23.537659 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:23.537660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:23.537872 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:23.537929 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:23.537931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.538702 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.538703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:23.539308 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:23.539310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.539786 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.539787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:23.539844 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:23.540470 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:23.540476 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:23.542947 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
W20260811 11:16:23.543215 6709 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:23.537334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:34365 (local address 127.28.25.62:37543)
0811 11:16:23.537427 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:23.537429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:23.537462 (+ 33us) server_negotiation.cc:407] Connection header received
0811 11:16:23.537549 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:23.537549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:23.537562 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:23.537631 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:23.538033 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.538591 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.539406 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.539610 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.539679 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:23.543131 (+ 3452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:34365: BlockingRecv error: recv got EOF from 127.28.25.2:34365 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 11 11:16:23 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:23 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:23.685639 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:23.680019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:43469)
0811 11:16:23.680196 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:23.680211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:23.680305 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:23.680498 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:23.680499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:23.680695 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:23.680750 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:23.680752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.681518 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.681519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:23.682146 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:23.682149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.682584 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.682585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:23.682649 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:23.683362 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:23.683369 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:23.685564 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
W20260811 11:16:23.685818 6709 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:23.680229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:43469 (local address 127.28.25.62:37543)
0811 11:16:23.680328 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:23.680330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:23.680362 (+ 32us) server_negotiation.cc:407] Connection header received
0811 11:16:23.680391 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:23.680392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:23.680407 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:23.680474 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:23.680841 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.681403 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.682244 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:23.682478 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:23.682580 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:23.685745 (+ 3165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:43469: BlockingRecv error: recv got EOF from 127.28.25.3:43469 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 11 11:16:24 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:24 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:24.403537 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:24.397957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:53771)
0811 11:16:24.398121 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:24.398138 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:24.398191 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:24.398436 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:24.398438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:24.398616 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:24.398673 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:24.398675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.399542 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.399543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:24.400115 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:24.400116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.400542 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.400543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:24.400590 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:24.401156 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:24.401161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:24.403463 (+ 2302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
W20260811 11:16:24.403721 6713 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:24.398058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:53771 (local address 127.28.25.62:37543)
0811 11:16:24.398252 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:24.398254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:24.398267 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:16:24.398294 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:24.398295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:24.398306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:24.398383 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:24.398770 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.399434 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.400216 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.400454 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.400524 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:24.403617 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:53771: BlockingRecv error: recv got EOF from 127.28.25.1:53771 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":120,"threads_started":1}
Aug 11 11:16:24 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:24 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:24.549110 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:24.543751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:54543)
0811 11:16:24.543915 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:24.543931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:24.543994 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:24.544175 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:24.544176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:24.544377 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:24.544436 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:24.544438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.545298 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.545299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:24.545870 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:24.545872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.546290 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.546292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:24.546342 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:24.546939 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:24.546944 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:24.549041 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
W20260811 11:16:24.549321 6713 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:24.543831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:54543 (local address 127.28.25.62:37543)
0811 11:16:24.543918 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:24.543921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:24.543997 (+ 76us) server_negotiation.cc:407] Connection header received
0811 11:16:24.544046 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:24.544047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:24.544058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:24.544120 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:24.544551 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.545143 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.545984 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.546199 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.546275 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:24.549229 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:54543: BlockingRecv error: recv got EOF from 127.28.25.2:54543 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 11 11:16:24 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:24 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:24.692219 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:24.686264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:38025)
0811 11:16:24.686532 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:24.686554 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:24.686654 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:24.686924 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:24.686925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:24.687176 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:24.687237 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:24.687239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.688007 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.688009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:24.688732 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:24.688736 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.689265 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.689266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:24.689333 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:24.689919 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:24.689924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:24.692111 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":170,"threads_started":1}
W20260811 11:16:24.692303 6713 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:24.686421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:38025 (local address 127.28.25.62:37543)
0811 11:16:24.686537 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:24.686539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:24.686640 (+ 101us) server_negotiation.cc:407] Connection header received
0811 11:16:24.686716 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:24.686717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:24.686731 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:24.686895 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:24.687334 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.687897 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.688929 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:24.689157 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:24.689269 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:24.692220 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:38025: BlockingRecv error: recv got EOF from 127.28.25.3:38025 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 11 11:16:25 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:25 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:25.413614 6716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:25.404139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:43323)
0811 11:16:25.404350 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:25.404363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:25.404453 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:25.404611 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:25.404612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:25.404879 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:25.404988 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:25.404991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.405975 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.405977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:25.406580 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:25.406582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.407020 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.407021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:25.407080 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:25.411368 (+ 4288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:25.411374 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:25.413543 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":115,"threads_started":1}
W20260811 11:16:25.413902 6717 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:25.404268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:43323 (local address 127.28.25.62:37543)
0811 11:16:25.404444 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:25.404446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:25.404457 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:16:25.404507 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:25.404508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:25.404519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:25.404586 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:25.405134 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.405882 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.406675 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.406903 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.406988 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:25.413823 (+ 6835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:43323: BlockingRecv error: recv got EOF from 127.28.25.1:43323 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":116,"threads_started":1}
Aug 11 11:16:25 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:25 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:25.555862 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:25.549702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:51461)
0811 11:16:25.549948 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:25.549960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:25.550036 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:25.550199 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:25.550200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:25.550374 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:25.550434 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:25.550436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.551247 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.551248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:25.552153 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:25.552157 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.552719 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.552720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:25.552779 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:25.553396 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:25.553401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:25.555796 (+ 2395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":116,"threads_started":1}
W20260811 11:16:25.555976 6717 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:25.549853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:51461 (local address 127.28.25.62:37543)
0811 11:16:25.549917 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:25.549919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:25.550077 (+ 158us) server_negotiation.cc:407] Connection header received
0811 11:16:25.550119 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:25.550120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:25.550132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:25.550177 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:25.550512 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.551073 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.552278 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.552616 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.552707 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:25.555872 (+ 3165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:51461: BlockingRecv error: recv got EOF from 127.28.25.2:51461 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 11 11:16:25 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:25 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:25.698395 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:25.692913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:46465)
0811 11:16:25.693087 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:25.693101 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:25.693178 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:25.693431 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:25.693433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:25.693617 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:25.693670 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:25.693672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.694428 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.694429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:25.695164 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:25.695166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.695555 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.695556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:25.695610 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:25.696120 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:25.696125 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:25.698269 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
W20260811 11:16:25.698411 6717 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:25.693057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:46465 (local address 127.28.25.62:37543)
0811 11:16:25.693163 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:25.693165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:25.693177 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:16:25.693286 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:25.693287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:25.693305 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:25.693375 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:25.693765 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.694315 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.695264 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:25.695460 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:25.695524 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:25.698354 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:46465: BlockingRecv error: recv got EOF from 127.28.25.3:46465 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 11 11:16:26 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:26 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:26.420013 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:26.414325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:59807)
0811 11:16:26.414514 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:26.414527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:26.414599 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:26.414949 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:26.414950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:26.415165 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:26.415257 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:26.415259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.416039 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.416040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:26.416612 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:26.416614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.417081 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.417083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:26.417137 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:26.417655 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:26.417661 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:26.419925 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":121,"threads_started":1}
W20260811 11:16:26.420148 6721 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:26.414541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:59807 (local address 127.28.25.62:37543)
0811 11:16:26.414675 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:26.414677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:26.414688 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:16:26.414730 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:26.414731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:26.414743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:26.414820 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:26.415360 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.415930 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.416783 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.416993 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.417056 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:26.420068 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:59807: BlockingRecv error: recv got EOF from 127.28.25.1:59807 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
Aug 11 11:16:26 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:26 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:26.561332 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:26.556475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:48925)
0811 11:16:26.556643 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:26.556654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:26.556706 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:26.556894 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:26.556895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:26.557083 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:26.557146 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:26.557148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.557870 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.557871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:26.558460 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:26.558462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.558803 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.558804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:26.558846 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:26.559328 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:26.559332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:26.561269 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
W20260811 11:16:26.561463 6721 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:26.556635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:48925 (local address 127.28.25.62:37543)
0811 11:16:26.556700 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:26.556701 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:26.556711 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:16:26.556773 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:26.556774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:26.556786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:26.556842 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:26.557251 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.557777 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.558539 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.558738 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.558793 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:26.561385 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:48925: BlockingRecv error: recv got EOF from 127.28.25.2:48925 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 11 11:16:26 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:26 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:26.704169 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:26.699046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:42255)
0811 11:16:26.699208 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:26.699220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:26.699299 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:26.699430 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:26.699431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:26.699606 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:26.699661 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:26.699662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.700325 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.700326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:26.700884 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:26.700886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.701244 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.701245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:26.701331 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:26.702024 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:26.702029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:26.704053 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
W20260811 11:16:26.704211 6721 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:26.699149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:42255 (local address 127.28.25.62:37543)
0811 11:16:26.699209 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:26.699211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:26.699292 (+ 81us) server_negotiation.cc:407] Connection header received
0811 11:16:26.699347 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:26.699347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:26.699360 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:26.699413 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:26.699725 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.700246 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.700973 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:26.701159 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:26.701210 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:26.704145 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:42255: BlockingRecv error: recv got EOF from 127.28.25.3:42255 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 11 11:16:27 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:27 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:27.425640 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:27.420625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:37759)
0811 11:16:27.420802 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:27.420814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:27.420880 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:27.421027 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:27.421027 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:27.421194 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:27.421248 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:27.421249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.421957 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.421959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:27.422511 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:27.422513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.422916 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.422917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:27.423022 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:27.423574 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:27.423579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:27.425566 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
W20260811 11:16:27.425779 6726 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:27.420747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:37759 (local address 127.28.25.62:37543)
0811 11:16:27.420858 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:27.420860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:27.420870 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:16:27.420938 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:27.420939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:27.420949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:27.421009 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:27.421318 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.421881 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.422592 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.422847 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.422904 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:27.425684 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:37759: BlockingRecv error: recv got EOF from 127.28.25.1:37759 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
Aug 11 11:16:27 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:27 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:27.567397 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:27.561970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:55239)
0811 11:16:27.562132 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:27.562148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:27.562229 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:27.562424 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:27.562425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:27.562675 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:27.562737 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:27.562740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.563700 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.563701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:27.564280 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:27.564283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.564729 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.564730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:27.564781 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:27.565269 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:27.565275 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:27.567308 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
W20260811 11:16:27.567469 6726 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:27.562104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:55239 (local address 127.28.25.62:37543)
0811 11:16:27.562164 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:27.562167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:27.562209 (+ 42us) server_negotiation.cc:407] Connection header received
0811 11:16:27.562349 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:27.562351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:27.562364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:27.562418 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:27.562901 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.563591 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.564415 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.564642 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.564713 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:27.567417 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:55239: BlockingRecv error: recv got EOF from 127.28.25.2:55239 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 11 11:16:27 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:27 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:27.710330 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:27.704911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:43955)
0811 11:16:27.705127 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:27.705138 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:27.705202 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:27.705365 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:27.705366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:27.705550 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:27.705607 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:27.705609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.706348 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.706349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:27.706925 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:27.706928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.707469 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.707470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:27.707545 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:27.708112 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:27.708126 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:27.710259 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
W20260811 11:16:27.710518 6726 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:27.705064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:43955 (local address 127.28.25.62:37543)
0811 11:16:27.705140 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:27.705142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:27.705202 (+ 60us) server_negotiation.cc:407] Connection header received
0811 11:16:27.705275 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:27.705276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:27.705289 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:27.705337 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:27.705690 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.706256 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.707067 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:27.707394 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:27.707467 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:27.710448 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:43955: BlockingRecv error: recv got EOF from 127.28.25.3:43955 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 11:16:28 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:28 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:28.431721 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:28.426208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:56887)
0811 11:16:28.426371 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:28.426382 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:28.426437 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:28.426639 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:28.426640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:28.426881 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:28.426943 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:28.426946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.427695 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.427696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:28.428282 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:28.428286 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.428642 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.428643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:28.428686 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:28.429332 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:28.429337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:28.431600 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
W20260811 11:16:28.431802 6730 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:28.426295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:56887 (local address 127.28.25.62:37543)
0811 11:16:28.426470 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:28.426471 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:28.426481 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:16:28.426507 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:28.426508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:28.426519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:28.426595 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:28.427056 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.427604 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.428363 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.428577 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.428633 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:28.431725 (+ 3092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:56887: BlockingRecv error: recv got EOF from 127.28.25.1:56887 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
Aug 11 11:16:28 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:28 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:28.573268 6731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:28.567952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:41121)
0811 11:16:28.568247 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:28.568260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:28.568366 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:28.568544 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:28.568545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:28.568745 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:28.568796 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:28.568798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.569506 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.569507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:28.570056 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:28.570057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.570446 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.570447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:28.570509 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:28.571105 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:28.571110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:28.573164 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":100,"threads_started":1}
W20260811 11:16:28.573326 6730 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:28.568056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:41121 (local address 127.28.25.62:37543)
0811 11:16:28.568243 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:28.568246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:28.568344 (+ 98us) server_negotiation.cc:407] Connection header received
0811 11:16:28.568454 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:28.568455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:28.568468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:28.568534 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:28.568884 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.569412 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.570151 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.570356 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.570446 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:28.573259 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:41121: BlockingRecv error: recv got EOF from 127.28.25.2:41121 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
Aug 11 11:16:28 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:28 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:28.716017 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:28.711007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:56777)
0811 11:16:28.711217 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:28.711229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:28.711311 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:28.711466 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:28.711467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:28.711644 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:28.711701 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:28.711703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.712387 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.712388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:28.712994 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:28.712997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.713362 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.713363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:28.713414 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:28.713901 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:28.713906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:28.715945 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":123,"threads_started":1}
W20260811 11:16:28.716163 6730 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:28.711063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:56777 (local address 127.28.25.62:37543)
0811 11:16:28.711121 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:28.711123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:28.711348 (+ 225us) server_negotiation.cc:407] Connection header received
0811 11:16:28.711375 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:28.711376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:28.711393 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:28.711436 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:28.711772 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.712280 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.713083 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:28.713268 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:28.713323 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:28.716061 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:56777: BlockingRecv error: recv got EOF from 127.28.25.3:56777 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 11 11:16:29 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:29 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:29.437294 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:29.432266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:39791)
0811 11:16:29.432452 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:29.432463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:29.432518 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:29.432675 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:29.432675 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:29.432850 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:29.432908 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:29.432910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.433632 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.433632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:29.434188 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:29.434190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.434601 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.434602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:29.434647 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:29.435262 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:29.435267 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:29.437230 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
W20260811 11:16:29.437510 6734 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:29.432405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:39791 (local address 127.28.25.62:37543)
0811 11:16:29.432544 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:29.432546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:29.432554 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:16:29.432579 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:29.432580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:29.432591 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:29.432660 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:29.432982 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.433554 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.434282 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.434526 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.434592 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:29.437365 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:39791: BlockingRecv error: recv got EOF from 127.28.25.1:39791 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
Aug 11 11:16:29 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:29 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:29.579447 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:29.573865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:60373)
0811 11:16:29.574037 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:29.574048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:29.574115 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:29.574271 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:29.574272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:29.574461 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:29.574514 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:29.574516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.575682 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.575683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:29.576262 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:29.576264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.576656 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.576657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:29.576727 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:29.577308 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:29.577313 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:29.579383 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
W20260811 11:16:29.579651 6734 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:29.573897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:60373 (local address 127.28.25.62:37543)
0811 11:16:29.573970 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:29.573972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:29.574114 (+ 142us) server_negotiation.cc:407] Connection header received
0811 11:16:29.574166 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:29.574167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:29.574181 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:29.574256 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:29.574607 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.575620 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.576348 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.576557 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.576618 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:29.579563 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:60373: BlockingRecv error: recv got EOF from 127.28.25.2:60373 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 11 11:16:29 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:29 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:29.721845 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:29.716621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:60417)
0811 11:16:29.716828 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:29.716839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:29.716905 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:29.717110 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:29.717111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:29.717296 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:29.717352 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:29.717354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.718026 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.718027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:29.718560 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:29.718561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.718885 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.718885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:29.718934 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:29.719548 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:29.719553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:29.721779 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
W20260811 11:16:29.721938 6734 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:29.716763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:60417 (local address 127.28.25.62:37543)
0811 11:16:29.716821 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:29.716823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:29.716890 (+ 67us) server_negotiation.cc:407] Connection header received
0811 11:16:29.716958 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:29.716958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:29.716971 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:29.717035 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:29.717445 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.717949 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.718634 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:29.718815 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:29.718866 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:29.721885 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:60417: BlockingRecv error: recv got EOF from 127.28.25.3:60417 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 11 11:16:30 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:30 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:30.447381 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:30.437939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:48735)
0811 11:16:30.438093 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:30.438105 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:30.438180 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:30.438359 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:30.438361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:30.438560 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:30.438617 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:30.438619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.439514 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.439516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:30.440073 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:30.440076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.440486 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.440487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:30.440538 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:30.444923 (+ 4385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:30.444930 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:30.447315 (+ 2385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
W20260811 11:16:30.447592 6739 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:30.438024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:48735 (local address 127.28.25.62:37543)
0811 11:16:30.438175 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:30.438177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:30.438189 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:16:30.438233 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:30.438233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:30.438244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:30.438310 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:30.438733 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.439397 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.440164 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.440404 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.440467 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:30.447516 (+ 7049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:48735: BlockingRecv error: recv got EOF from 127.28.25.1:48735 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":57,"threads_started":1}
Aug 11 11:16:30 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:30 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:30.585187 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:30.580075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:53225)
0811 11:16:30.580261 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:30.580276 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:30.580344 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:30.580499 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:30.580500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:30.580680 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:30.580735 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:30.580736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.581470 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.581471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:30.582199 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:30.582202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.582561 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.582562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:30.582605 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:30.583157 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:30.583162 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:30.585125 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":123,"threads_started":1}
W20260811 11:16:30.585321 6739 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:30.580158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:53225 (local address 127.28.25.62:37543)
0811 11:16:30.580243 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:30.580245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:30.580344 (+ 99us) server_negotiation.cc:407] Connection header received
0811 11:16:30.580399 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:30.580400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:30.580413 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:30.580479 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:30.580804 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.581360 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.582271 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.582490 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.582550 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:30.585237 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:53225: BlockingRecv error: recv got EOF from 127.28.25.2:53225 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 11 11:16:30 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:30 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:30.728302 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:30.722628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:51753)
0811 11:16:30.722760 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:30.722780 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:30.722837 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:30.723064 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:30.723065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:30.723234 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:30.723288 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:30.723290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.724122 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.724123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:30.724884 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:30.724920 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.725324 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.725325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:30.725391 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:30.725972 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:30.725977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:30.728196 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
W20260811 11:16:30.728358 6739 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:30.722738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:51753 (local address 127.28.25.62:37543)
0811 11:16:30.722806 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:30.722808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:30.722828 (+ 20us) server_negotiation.cc:407] Connection header received
0811 11:16:30.722904 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:30.722905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:30.722919 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:30.723057 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:30.723357 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.724010 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.725025 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:30.725242 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:30.725322 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:30.728288 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:51753: BlockingRecv error: recv got EOF from 127.28.25.3:51753 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 11 11:16:31 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:31 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:31.452981 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:31.448032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:48129)
0811 11:16:31.448190 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:31.448201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:31.448271 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:31.448539 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:31.448540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:31.448714 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:31.448766 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:31.448769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.449474 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.449475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:31.450035 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:31.450037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.450427 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.450428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:31.450470 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:31.451056 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:31.451061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:31.452917 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
W20260811 11:16:31.453203 6743 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:31.448248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:48129 (local address 127.28.25.62:37543)
0811 11:16:31.448407 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:31.448409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:31.448417 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:16:31.448439 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:31.448440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:31.448450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:31.448517 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:31.448857 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.449386 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.450124 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.450334 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.450392 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:31.453104 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:48129: BlockingRecv error: recv got EOF from 127.28.25.1:48129 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
Aug 11 11:16:31 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:31 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:31.591292 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:31.585876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:57535)
0811 11:16:31.586104 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:31.586118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:31.586220 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:31.586441 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:31.586442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:31.586636 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:31.586688 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:31.586690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.587515 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.587516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:31.588070 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:31.588072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.588475 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.588476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:31.588538 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:31.589055 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:31.589060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:31.591211 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":113,"threads_started":1}
W20260811 11:16:31.591683 6743 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:31.585993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:57535 (local address 127.28.25.62:37543)
0811 11:16:31.586091 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:31.586094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:31.586197 (+ 103us) server_negotiation.cc:407] Connection header received
0811 11:16:31.586306 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:31.586307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:31.586325 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:31.586401 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:31.586773 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.587419 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.588178 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.588395 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.588474 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:31.591614 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:57535: BlockingRecv error: recv got EOF from 127.28.25.2:57535 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 11 11:16:31 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:31 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:31.734457 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:31.728832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:53285)
0811 11:16:31.729032 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:31.729064 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:31.729151 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:31.729386 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:31.729388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:31.729668 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:31.729857 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:31.729862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.730633 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.730635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:31.731293 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:31.731296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.731693 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.731694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:31.731767 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:31.732314 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:31.732319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:31.734381 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
W20260811 11:16:31.734648 6743 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:31.729020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:53285 (local address 127.28.25.62:37543)
0811 11:16:31.729090 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:31.729092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:31.729126 (+ 34us) server_negotiation.cc:407] Connection header received
0811 11:16:31.729236 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:31.729237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:31.729259 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:31.729359 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:31.729965 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.730509 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.731394 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:31.731599 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:31.731690 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:31.734556 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:53285: BlockingRecv error: recv got EOF from 127.28.25.3:53285 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 11 11:16:32 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:32 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:32.459071 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:32.453612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:55615)
0811 11:16:32.453812 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:32.453827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:32.453916 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:32.454075 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:32.454076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:32.454257 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:32.454308 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:32.454310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.455175 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.455176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:32.455739 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:32.455740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.456231 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.456232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:32.456281 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:32.456855 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:32.456861 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:32.458938 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
W20260811 11:16:32.459233 6747 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:32.453750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:55615 (local address 127.28.25.62:37543)
0811 11:16:32.453902 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:32.453904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:32.453915 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:16:32.453972 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:32.453973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:32.453985 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:32.454052 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:32.454404 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.455088 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.455924 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.456147 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.456214 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:32.459137 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:55615: BlockingRecv error: recv got EOF from 127.28.25.1:55615 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
Aug 11 11:16:32 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:32 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:32.597510 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:32.591929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:60195)
0811 11:16:32.592088 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:32.592099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:32.592187 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:32.592410 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:32.592412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:32.592655 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:32.592718 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:32.592721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.593443 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.593444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:32.594033 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:32.594035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.594466 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.594467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:32.594518 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:32.595173 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:32.595179 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:32.597407 (+ 2228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
W20260811 11:16:32.597605 6747 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:32.592111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:60195 (local address 127.28.25.62:37543)
0811 11:16:32.592173 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:32.592177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:32.592193 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:16:32.592293 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:32.592294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:32.592310 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:32.592378 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:32.592845 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.593356 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.594139 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.594381 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.594446 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:32.597531 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:60195: BlockingRecv error: recv got EOF from 127.28.25.2:60195 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 11 11:16:32 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:32 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:32.740669 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:32.735099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:39991)
0811 11:16:32.735442 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:32.735455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:32.735538 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:32.735702 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:32.735702 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:32.735898 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:32.735961 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:32.735962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.736652 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.736653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:32.737229 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:32.737230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.737674 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.737676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:32.737737 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:32.738333 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:32.738338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:32.740603 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":237,"threads_started":1}
W20260811 11:16:32.740830 6747 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:32.735300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:39991 (local address 127.28.25.62:37543)
0811 11:16:32.735399 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:32.735402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:32.735538 (+ 136us) server_negotiation.cc:407] Connection header received
0811 11:16:32.735589 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:32.735590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:32.735603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:32.735669 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:32.736042 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.736565 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.737318 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:32.737544 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:32.737607 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:32.740755 (+ 3148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:39991: BlockingRecv error: recv got EOF from 127.28.25.3:39991 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 11 11:16:33 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:33 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:33.465374 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:33.459702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:47783)
0811 11:16:33.459987 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:33.460001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:33.460070 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:33.460231 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:33.460231 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:33.460438 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:33.460491 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:33.460493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.461237 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.461239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:33.462131 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:33.462136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.462631 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.462632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:33.462704 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:33.463293 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:33.463298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:33.465267 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":153,"threads_started":1}
W20260811 11:16:33.465448 6751 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:33.459902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:47783 (local address 127.28.25.62:37543)
0811 11:16:33.460048 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:33.460050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:33.460062 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:16:33.460127 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:33.460128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:33.460139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:33.460203 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:33.460584 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.461121 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.462233 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.462526 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.462615 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:33.465371 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:47783: BlockingRecv error: recv got EOF from 127.28.25.1:47783 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
Aug 11 11:16:33 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:33 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:33.604055 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:33.598272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:50001)
0811 11:16:33.598428 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:33.598439 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:33.598505 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:33.598644 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:33.598645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:33.598831 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:33.598886 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:33.598887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.599771 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.599773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:33.600387 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:33.600390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.600785 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.600786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:33.600841 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:33.601523 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:33.601528 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:33.603894 (+ 2366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
W20260811 11:16:33.604081 6751 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:33.598340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:50001 (local address 127.28.25.62:37543)
0811 11:16:33.598408 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:33.598410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:33.598493 (+ 83us) server_negotiation.cc:407] Connection header received
0811 11:16:33.598581 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:33.598582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:33.598593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:33.598639 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:33.598990 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.599646 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.600487 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.600682 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.600738 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:33.604025 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:50001: BlockingRecv error: recv got EOF from 127.28.25.2:50001 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 11 11:16:33 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:33 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:33.746490 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:33.741269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:47823)
0811 11:16:33.741540 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:33.741553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:33.741660 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:33.741846 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:33.741847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:33.742066 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:33.742117 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:33.742119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.742826 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.742828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:33.743478 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:33.743482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.743871 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.743872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:33.743935 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:33.744427 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:33.744432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:33.746363 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":143,"threads_started":1}
W20260811 11:16:33.746531 6751 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:33.741426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:47823 (local address 127.28.25.62:37543)
0811 11:16:33.741533 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:33.741536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:33.741637 (+ 101us) server_negotiation.cc:407] Connection header received
0811 11:16:33.741733 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:33.741734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:33.741757 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:33.741838 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:33.742209 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.742732 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.743578 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:33.743778 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:33.743869 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:33.746459 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:47823: BlockingRecv error: recv got EOF from 127.28.25.3:47823 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 11 11:16:34 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:34 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:34.471581 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:34.466004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:52955)
0811 11:16:34.466259 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:34.466271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:34.466352 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:34.466530 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:34.466531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:34.466713 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:34.466765 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:34.466767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.467718 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.467720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:34.468340 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:34.468343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.468756 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.468757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:34.468865 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:34.469332 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:34.469337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:34.471518 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":170,"threads_started":1}
W20260811 11:16:34.471750 6755 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:34.466152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:52955 (local address 127.28.25.62:37543)
0811 11:16:34.466273 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:34.466275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:34.466358 (+ 83us) server_negotiation.cc:407] Connection header received
0811 11:16:34.466429 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:34.466429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:34.466441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:34.466508 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:34.466891 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.467608 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.468459 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.468663 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.468723 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:34.471645 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:52955: BlockingRecv error: recv got EOF from 127.28.25.1:52955 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
Aug 11 11:16:34 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:34 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:34.609978 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:34.604746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:49043)
0811 11:16:34.604982 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:34.604994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:34.605091 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:34.605249 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:34.605250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:34.605455 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:34.605506 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:34.605507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.606252 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.606253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:34.606856 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:34.606858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.607281 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.607282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:34.607339 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:34.607900 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:34.607906 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:34.609870 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":73,"threads_started":1}
W20260811 11:16:34.610049 6755 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:34.605006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:49043 (local address 127.28.25.62:37543)
0811 11:16:34.605088 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:34.605090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:34.605101 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:16:34.605154 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:34.605155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:34.605174 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:34.605240 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:34.605611 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.606146 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.606985 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.607199 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.607282 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:34.609985 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:49043: BlockingRecv error: recv got EOF from 127.28.25.2:49043 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 11 11:16:34 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:34 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:34.754113 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:34.747252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:39895)
0811 11:16:34.747441 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:34.747453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:34.747522 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:34.747724 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:34.747725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:34.747915 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:34.747979 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:34.747981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.748972 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.748974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:34.749828 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:34.749833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.750455 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.750456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:34.750545 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:34.751242 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:34.751249 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:34.754001 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":121,"threads_started":1}
W20260811 11:16:34.754249 6755 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:34.747344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:39895 (local address 127.28.25.62:37543)
0811 11:16:34.747449 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:34.747452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:34.747520 (+ 68us) server_negotiation.cc:407] Connection header received
0811 11:16:34.747576 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:34.747577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:34.747593 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:34.747669 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:34.748090 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.748845 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.749945 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:34.750313 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:34.750402 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:34.754170 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:39895: BlockingRecv error: recv got EOF from 127.28.25.3:39895 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 11 11:16:35 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:35 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:35.482663 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:35.472306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:39807)
0811 11:16:35.472434 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:35.472453 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:35.472507 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:35.472749 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:35.472750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:35.473002 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:35.473072 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:35.473076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.473968 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.473969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:35.474725 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:35.474728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.475232 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.475233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:35.475284 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:35.480305 (+ 5021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:35.480313 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:35.482579 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
W20260811 11:16:35.482944 6759 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:35.472390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:39807 (local address 127.28.25.62:37543)
0811 11:16:35.472546 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:35.472548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:35.472558 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:16:35.472584 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:35.472585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:35.472604 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:35.472716 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:35.473171 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.473852 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.474870 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.475138 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.475262 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:35.482868 (+ 7606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:39807: BlockingRecv error: recv got EOF from 127.28.25.1:39807 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
Aug 11 11:16:35 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:35 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:35.616459 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:35.610632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:60085)
0811 11:16:35.610854 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:35.610877 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:35.610975 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:35.611139 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:35.611140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:35.611350 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:35.611441 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:35.611442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.612194 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.612196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:35.612862 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:35.612864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.613299 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.613300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:35.613358 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:35.613925 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:35.613931 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:35.616346 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":116,"threads_started":1}
W20260811 11:16:35.616614 6759 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:35.610802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:60085 (local address 127.28.25.62:37543)
0811 11:16:35.610861 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:35.610864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:35.610942 (+ 78us) server_negotiation.cc:407] Connection header received
0811 11:16:35.611037 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:35.611038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:35.611050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:35.611104 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:35.611545 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.612102 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.612983 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.613202 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.613270 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:35.616488 (+ 3218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:60085: BlockingRecv error: recv got EOF from 127.28.25.2:60085 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 11 11:16:35 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:35 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.3@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:35.760331 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:35.754883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.3:44071)
0811 11:16:35.755093 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:35.755108 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:35.755171 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:35.755383 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:35.755384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:35.755572 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:35.755629 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:35.755631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.756489 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.756491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:35.757132 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:35.757135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.757557 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.757558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:35.757613 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:35.758152 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:35.758157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:35.760248 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":147,"threads_started":1}
W20260811 11:16:35.760416 6759 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:35.755086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:44071 (local address 127.28.25.62:37543)
0811 11:16:35.755163 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:35.755165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:35.755178 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:16:35.755222 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:35.755223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:35.755237 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:35.755304 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:35.755801 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.756356 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.757238 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:35.757451 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:35.757514 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:35.760342 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.3:44071: BlockingRecv error: recv got EOF from 127.28.25.3:44071 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 11:16:36 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:36 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.1@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:36.489066 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:36.483366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.1:45053)
0811 11:16:36.483523 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:36.483535 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:36.483589 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:36.483738 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:36.483739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:36.483970 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:36.484024 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:36.484026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:36.484898 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:36.484900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:36.485530 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:36.485533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:36.486070 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:36.486071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:36.486126 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:36.486754 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:36.486789 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:36.488984 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
W20260811 11:16:36.489187 6763 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:36.483450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:45053 (local address 127.28.25.62:37543)
0811 11:16:36.483597 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:36.483599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:36.483610 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:16:36.483637 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:36.483638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:36.483648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:36.483723 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:36.484143 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:36.484722 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:36.485703 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:36.485968 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:36.486041 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:36.489071 (+ 3030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.1:45053: BlockingRecv error: recv got EOF from 127.28.25.1:45053 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
Aug 11 11:16:36 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:16:36 dist-test-slave-2w3w krb5kdc[5572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.25.2@KRBTEST.COM for oryx/127.28.25.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:16:36.622085 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:36.617151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37543 (local address 127.28.25.2:34419)
0811 11:16:36.617369 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:36.617389 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:36.617464 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:36.617618 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:36.617619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:36.617798 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:36.617853 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:36.617854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:36.618555 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:36.618556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:36.619139 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:36.619141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:36.619494 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:36.619496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:36.619551 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:36.620163 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:36.620168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:36.622005 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.25.62:37543: Server oryx/127.28.25.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":103,"threads_started":1}
W20260811 11:16:36.622320 6763 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:36.617314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:34419 (local address 127.28.25.62:37543)
0811 11:16:36.617421 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:36.617423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:36.617446 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:16:36.617536 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:36.617537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:36.617548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:36.617592 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:36.617942 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:36.618467 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:36.619210 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:36.619419 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:36.619472 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:36.622245 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.25.2:34419: BlockingRecv error: recv got EOF from 127.28.25.2:34419 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 11:16:36.656390 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 6230
I20260811 11:16:36.664692 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 6405
I20260811 11:16:36.675498 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 6555
I20260811 11:16:36.684170 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 6154
2026-08-11T11:16:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17843 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-11T11:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:16:36Z Disabled control of system clock
I20260811 11:16:36.719550 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:46799
--webserver_interface=127.28.25.62
--webserver_port=0
--builtin_ntp_servers=127.28.25.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 11:16:36.807900 6769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:36.808137 6769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:36.808161 6769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:36.809516 6769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:16:36.809576 6769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:36.809589 6769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:36.809602 6769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:16:36.809614 6769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:16:36.811204 6769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:46799
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:46799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:36.811527 6769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:36.811899 6769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:36.817200 6774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:36.817247 6777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:36.817557 6769 server_base.cc:1034] running on GCE node
W20260811 11:16:36.817253 6775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:36.817955 6769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:36.818251 6769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:36.819446 6769 hybrid_clock.cc:648] HybridClock initialized: now 1786446996819429 us; error 32 us; skew 500 ppm
I20260811 11:16:36.821168 6769 webserver.cc:460] Webserver started at http://127.28.25.62:39245/ using document root <none> and password file <none>
I20260811 11:16:36.821458 6769 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:36.821514 6769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:36.821698 6769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:36.822926 6769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "23222076d14040ba853592131a5d8fb7"
format_stamp: "Formatted at 2026-08-11 11:16:36 on dist-test-slave-2w3w"
server_key: "b3d47d6142f0702f503ffa737dea1587"
server_key_iv: "706daaf96cd2e33001bcc230ccc88379"
server_key_version: "encryptionkey@0"
I20260811 11:16:36.823412 6769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "23222076d14040ba853592131a5d8fb7"
format_stamp: "Formatted at 2026-08-11 11:16:36 on dist-test-slave-2w3w"
server_key: "b3d47d6142f0702f503ffa737dea1587"
server_key_iv: "706daaf96cd2e33001bcc230ccc88379"
server_key_version: "encryptionkey@0"
I20260811 11:16:36.824985 6769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 11:16:36.825876 6783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:36.826100 6769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:16:36.826200 6769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "23222076d14040ba853592131a5d8fb7"
format_stamp: "Formatted at 2026-08-11 11:16:36 on dist-test-slave-2w3w"
server_key: "b3d47d6142f0702f503ffa737dea1587"
server_key_iv: "706daaf96cd2e33001bcc230ccc88379"
server_key_version: "encryptionkey@0"
I20260811 11:16:36.826287 6769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:36.839026 6769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:36.839435 6769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:36.839604 6769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:36.844244 6769 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:46799
I20260811 11:16:36.844395 6841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:46799 every 8 connection(s)
I20260811 11:16:36.844758 6769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:16:36.845600 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 6769
I20260811 11:16:36.845603 6842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:36.845687 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:16:36.845916 28772 external_mini_cluster.cc:1428] Setting key 99fe574b68da5a057a15d05957c03fad
I20260811 11:16:36.849375 6842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7: Bootstrap starting.
I20260811 11:16:36.850086 6842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:36.850451 6842 log.cc:824] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:36.851368 6842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7: No bootstrap required, opened a new log
I20260811 11:16:36.851917 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:36.846589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.28.25.62:46799)
0811 11:16:36.846831 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:36.846839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:36.846867 (+ 28us) server_negotiation.cc:407] Connection header received
0811 11:16:36.847378 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:36.847386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:36.847467 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:36.847633 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:36.847873 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:36.848785 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:36.849665 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:36.850109 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:36.850251 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:36.851126 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:36.851130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:36.851130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:16:36.851321 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:36.851359 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:16:36.851371 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:36.851456 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:36.851622 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 11:16:36.851728 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260811 11:16:36.853753 6842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23222076d14040ba853592131a5d8fb7" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46799 } }
I20260811 11:16:36.853935 6842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:36.853961 6842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23222076d14040ba853592131a5d8fb7, State: Initialized, Role: FOLLOWER
I20260811 11:16:36.854110 6842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23222076d14040ba853592131a5d8fb7" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46799 } }
I20260811 11:16:36.854214 6842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:16:36.854242 6842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:16:36.854316 6842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:36.855105 6842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23222076d14040ba853592131a5d8fb7" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46799 } }
I20260811 11:16:36.855244 6842 leader_election.cc:302] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23222076d14040ba853592131a5d8fb7; no voters:
I20260811 11:16:36.855450 6842 leader_election.cc:288] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:16:36.855540 6848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:36.855733 6848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [term 1 LEADER]: Becoming Leader. State: Replica: 23222076d14040ba853592131a5d8fb7, State: Running, Role: LEADER
I20260811 11:16:36.855791 6842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:16:36.855836 6848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23222076d14040ba853592131a5d8fb7" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46799 } }
I20260811 11:16:36.856248 6850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23222076d14040ba853592131a5d8fb7. Latest consensus state: current_term: 1 leader_uuid: "23222076d14040ba853592131a5d8fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23222076d14040ba853592131a5d8fb7" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46799 } } }
I20260811 11:16:36.856345 6850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:36.856235 6849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23222076d14040ba853592131a5d8fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23222076d14040ba853592131a5d8fb7" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46799 } } }
I20260811 11:16:36.856540 6849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:36.856662 6855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:16:36.857190 6855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:16:36.858820 6855 catalog_manager.cc:1347] Generated new cluster ID: df200c16c5474ba798d76147d7cac826
I20260811 11:16:36.858896 6855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:16:36.869843 6855 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:16:36.870483 6855 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:16:36.873041 6855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23222076d14040ba853592131a5d8fb7: Generated new TSK 0
I20260811 11:16:36.873263 6855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:16:36.878988 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46799
--builtin_ntp_servers=127.28.25.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260811 11:16:36.967355 6867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:36.967594 6867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:36.967620 6867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:36.968969 6867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:36.969033 6867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:36.969061 6867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:16:36.970570 6867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:36.970882 6867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:36.971251 6867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:36.976819 6874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:36.976846 6873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:36.976814 6877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:36.977339 6867 server_base.cc:1034] running on GCE node
I20260811 11:16:36.977559 6867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:36.977847 6867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:36.979044 6867 hybrid_clock.cc:648] HybridClock initialized: now 1786446996979023 us; error 33 us; skew 500 ppm
I20260811 11:16:36.980713 6867 webserver.cc:460] Webserver started at http://127.28.25.1:33831/ using document root <none> and password file <none>
I20260811 11:16:36.981002 6867 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:36.981065 6867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:36.981192 6867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:36.982735 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "3e0f49afe47f41df9e576c25e15ea850"
format_stamp: "Formatted at 2026-08-11 11:16:36 on dist-test-slave-2w3w"
server_key: "5ed6b142ee4a2edeef710c111803871f"
server_key_iv: "3f3d72477695e3a9f67ac89062165114"
server_key_version: "encryptionkey@0"
I20260811 11:16:36.983289 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "3e0f49afe47f41df9e576c25e15ea850"
format_stamp: "Formatted at 2026-08-11 11:16:36 on dist-test-slave-2w3w"
server_key: "5ed6b142ee4a2edeef710c111803871f"
server_key_iv: "3f3d72477695e3a9f67ac89062165114"
server_key_version: "encryptionkey@0"
I20260811 11:16:36.985159 6867 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 11:16:36.986279 6883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:36.986598 6867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:36.986718 6867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "3e0f49afe47f41df9e576c25e15ea850"
format_stamp: "Formatted at 2026-08-11 11:16:36 on dist-test-slave-2w3w"
server_key: "5ed6b142ee4a2edeef710c111803871f"
server_key_iv: "3f3d72477695e3a9f67ac89062165114"
server_key_version: "encryptionkey@0"
I20260811 11:16:36.986825 6867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:37.003095 6867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:37.003441 6867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:37.003562 6867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:37.003775 6867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:37.004170 6867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:37.004208 6867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.004233 6867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:37.004245 6867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.011175 6867 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:39047
I20260811 11:16:37.011390 7008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:39047 every 8 connection(s)
I20260811 11:16:37.011752 6867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:16:37.015875 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 6867
I20260811 11:16:37.015972 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:16:37.016193 28772 external_mini_cluster.cc:1428] Setting key 74fc9b68c46004f4c55b263b3229ad35
I20260811 11:16:37.017449 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46799
--builtin_ntp_servers=127.28.25.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 11:16:37.018313 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.012608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:36583 (local address 127.28.25.62:46799)
0811 11:16:37.012703 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:37.012705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:37.013023 (+ 318us) server_negotiation.cc:407] Connection header received
0811 11:16:37.013671 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:37.013672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:37.013688 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:37.013754 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:37.014478 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.015131 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.016400 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.016678 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.016768 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:37.017951 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:37.017953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:37.017953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:16:37.017960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:37.017986 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:16:37.017990 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:37.018067 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:37.018165 (+ 98us) server_negotiation.cc:299] Negotiation successful
0811 11:16:37.018221 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 11:16:37.018607 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.012594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46799 (local address 127.28.25.1:36583)
0811 11:16:37.012829 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:37.012885 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:37.013555 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:37.013829 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:37.013833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:37.013842 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:37.014367 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:37.014372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.015267 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.015268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:37.016302 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:37.016306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.016801 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.016803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:37.016990 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:37.017747 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:16:37.017748 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:37.017795 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:16:37.017832 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:37.018071 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:37.018079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:37.018083 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:16:37.018160 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 11:16:37.018315 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
I20260811 11:16:37.019235 7009 heartbeater.cc:347] Connected to a master server at 127.28.25.62:46799
I20260811 11:16:37.019409 7009 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:37.019644 7009 heartbeater.cc:511] Master 127.28.25.62:46799 requested a full tablet report, sending...
I20260811 11:16:37.020275 6802 ts_manager.cc:194] Registered new tserver with Master: 3e0f49afe47f41df9e576c25e15ea850 (127.28.25.1:39047)
I20260811 11:16:37.021014 6802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.1:36583
W20260811 11:16:37.110076 7012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:37.110320 7012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:37.110345 7012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:37.111768 7012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:37.111828 7012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:37.111858 7012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:16:37.113407 7012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:37.113732 7012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:37.114105 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:37.119737 7017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:37.119737 7018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:37.120003 7020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:37.120059 7012 server_base.cc:1034] running on GCE node
I20260811 11:16:37.120304 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:37.120637 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:37.121819 7012 hybrid_clock.cc:648] HybridClock initialized: now 1786446997121803 us; error 43 us; skew 500 ppm
I20260811 11:16:37.123468 7012 webserver.cc:460] Webserver started at http://127.28.25.2:36817/ using document root <none> and password file <none>
I20260811 11:16:37.123765 7012 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:37.123845 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:37.124027 7012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:37.125197 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "db8dcd47d3654bd0a352c5a8ae4e3662"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "f5f4ade91a2a4b1950aa5395b6f3714a"
server_key_iv: "ebdcc16387e44dcf2cea794c74de20cb"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.125610 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "db8dcd47d3654bd0a352c5a8ae4e3662"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "f5f4ade91a2a4b1950aa5395b6f3714a"
server_key_iv: "ebdcc16387e44dcf2cea794c74de20cb"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.127187 7012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260811 11:16:37.128139 7026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.128363 7012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:37.128505 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "db8dcd47d3654bd0a352c5a8ae4e3662"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "f5f4ade91a2a4b1950aa5395b6f3714a"
server_key_iv: "ebdcc16387e44dcf2cea794c74de20cb"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.128594 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:37.151196 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:37.151592 7012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:37.151772 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:37.152031 7012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:37.152446 7012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:37.152508 7012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.152545 7012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:37.152593 7012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.159703 7012 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:42363
I20260811 11:16:37.159752 7152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:42363 every 8 connection(s)
I20260811 11:16:37.160137 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:16:37.164623 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 7012
I20260811 11:16:37.164736 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:16:37.164948 28772 external_mini_cluster.cc:1428] Setting key dfde87c3300061337a8079bf9cd95b60
I20260811 11:16:37.166141 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46799
--builtin_ntp_servers=127.28.25.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 11:16:37.166440 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.160978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:39089 (local address 127.28.25.62:46799)
0811 11:16:37.161025 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:37.161027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:37.161328 (+ 301us) server_negotiation.cc:407] Connection header received
0811 11:16:37.161975 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:37.161976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:37.161990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:37.162054 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:37.162796 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.163429 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.164598 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.164919 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.165003 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:37.166146 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:37.166149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:37.166149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:16:37.166155 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:37.166172 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:16:37.166176 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:37.166227 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:37.166325 (+ 98us) server_negotiation.cc:299] Negotiation successful
0811 11:16:37.166358 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 11:16:37.166707 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.160957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46799 (local address 127.28.25.2:39089)
0811 11:16:37.161158 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:37.161197 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:37.161836 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:37.162115 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:37.162119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:37.162129 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:37.162686 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:37.162691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.163567 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.163568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:37.164478 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:37.164481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.165122 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.165124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:37.165318 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:37.165982 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:16:37.165983 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:37.166017 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:16:37.166044 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:37.166228 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:37.166235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:37.166239 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:16:37.166328 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 11:16:37.166454 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260811 11:16:37.167320 7153 heartbeater.cc:347] Connected to a master server at 127.28.25.62:46799
I20260811 11:16:37.167461 7153 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:37.167707 7153 heartbeater.cc:511] Master 127.28.25.62:46799 requested a full tablet report, sending...
I20260811 11:16:37.168198 6802 ts_manager.cc:194] Registered new tserver with Master: db8dcd47d3654bd0a352c5a8ae4e3662 (127.28.25.2:42363)
I20260811 11:16:37.168633 6802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.2:39089
W20260811 11:16:37.259471 7156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:37.259742 7156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:37.259784 7156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:37.261099 7156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:37.261178 7156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:37.261224 7156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:16:37.262801 7156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:37.263152 7156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:37.263513 7156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:37.269470 7162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:37.269481 7164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:37.269562 7156 server_base.cc:1034] running on GCE node
W20260811 11:16:37.269464 7161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:37.270004 7156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:37.270364 7156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:37.271569 7156 hybrid_clock.cc:648] HybridClock initialized: now 1786446997271545 us; error 44 us; skew 500 ppm
I20260811 11:16:37.273134 7156 webserver.cc:460] Webserver started at http://127.28.25.3:32961/ using document root <none> and password file <none>
I20260811 11:16:37.273430 7156 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:37.273509 7156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:37.273679 7156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:37.275061 7156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "f0170fa7da4c4acba0da97ba331776a3"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "a9ba77a7c2608d75f5543d00e4d17037"
server_key_iv: "89addb36da8513372209ebf8de5c4404"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.275476 7156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "f0170fa7da4c4acba0da97ba331776a3"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "a9ba77a7c2608d75f5543d00e4d17037"
server_key_iv: "89addb36da8513372209ebf8de5c4404"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.277029 7156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 11:16:37.278019 7170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.278261 7156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:16:37.278390 7156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "f0170fa7da4c4acba0da97ba331776a3"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "a9ba77a7c2608d75f5543d00e4d17037"
server_key_iv: "89addb36da8513372209ebf8de5c4404"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.278481 7156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:37.293596 7156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:37.293979 7156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:37.294147 7156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:37.294409 7156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:37.294812 7156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:37.294873 7156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.294943 7156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:37.294997 7156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.301651 7156 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:33343
I20260811 11:16:37.301862 7293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:33343 every 8 connection(s)
I20260811 11:16:37.302163 7156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:16:37.303159 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 7156
I20260811 11:16:37.303261 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:16:37.303504 28772 external_mini_cluster.cc:1428] Setting key 83905d8de84aa75fdf7e172acefb5a1d
I20260811 11:16:37.308115 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.303057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:33435 (local address 127.28.25.62:46799)
0811 11:16:37.303125 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:37.303128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:37.303389 (+ 261us) server_negotiation.cc:407] Connection header received
0811 11:16:37.304122 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:37.304123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:37.304133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:37.304200 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:37.305068 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.305566 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.306645 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.306851 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.306924 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:37.307896 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:37.307898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:37.307898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:16:37.307904 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:37.307921 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:16:37.307924 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:37.307970 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:37.308024 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 11:16:37.308051 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 11:16:37.308439 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.302938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46799 (local address 127.28.25.3:33435)
0811 11:16:37.303234 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:37.303289 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:37.304002 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:37.304305 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:37.304310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:37.304320 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:37.304977 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:37.304983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.305674 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.305675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:37.306539 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:37.306541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:37.307014 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:37.307016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:37.307181 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:37.307745 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:16:37.307745 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:37.307777 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:16:37.307804 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:37.307971 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:37.307974 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:37.307977 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:16:37.308025 (+ 48us) client_negotiation.cc:240] Negotiation successful
0811 11:16:37.308114 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
I20260811 11:16:37.309026 7294 heartbeater.cc:347] Connected to a master server at 127.28.25.62:46799
I20260811 11:16:37.309146 7294 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:37.309351 7294 heartbeater.cc:511] Master 127.28.25.62:46799 requested a full tablet report, sending...
I20260811 11:16:37.309854 6802 ts_manager.cc:194] Registered new tserver with Master: f0170fa7da4c4acba0da97ba331776a3 (127.28.25.3:33343)
I20260811 11:16:37.310276 6802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.3:33435
I20260811 11:16:37.315930 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 11:16:37.318204 6846 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:37.317524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.28.25.62:46799)
0811 11:16:37.317594 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:37.317596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:37.317767 (+ 171us) server_negotiation.cc:407] Connection header received
0811 11:16:37.317856 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:37.317856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:37.317866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:37.317931 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:37.318106 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58568: BlockingRecv error: recv got EOF from 127.0.0.1:58568 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:16:37.318341 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 6867
I20260811 11:16:37.328321 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 7012
I20260811 11:16:37.337203 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 7156
I20260811 11:16:37.349660 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 6769
2026-08-11T11:16:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (663 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-11T11:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:16:37Z Disabled control of system clock
I20260811 11:16:37.382500 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:42719
--webserver_interface=127.28.25.62
--webserver_port=0
--builtin_ntp_servers=127.28.25.20:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:42719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 11:16:37.474100 7307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:37.474365 7307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:37.474391 7307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:37.475816 7307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:16:37.475875 7307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:37.475889 7307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:37.475903 7307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:16:37.475916 7307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:16:37.477977 7307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:42719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:37.478331 7307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:37.478684 7307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:16:37.484392 7307 server_base.cc:1034] running on GCE node
W20260811 11:16:37.484346 7313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:37.484346 7312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:37.484370 7316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:37.484987 7307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:37.485390 7307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:37.486750 7307 hybrid_clock.cc:648] HybridClock initialized: now 1786446997486722 us; error 42 us; skew 500 ppm
I20260811 11:16:37.488615 7307 webserver.cc:460] Webserver started at http://127.28.25.62:40471/ using document root <none> and password file <none>
I20260811 11:16:37.488901 7307 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:37.488983 7307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:37.489161 7307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:37.490350 7307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "39cfebd5efe84366873779b8a05b58b8"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "1a15f8b3d18ff0d2fe993cd1fdf5aa15"
server_key_iv: "fb7f439fcdae6208b2d87975a8620ee5"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.490752 7307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "39cfebd5efe84366873779b8a05b58b8"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "1a15f8b3d18ff0d2fe993cd1fdf5aa15"
server_key_iv: "fb7f439fcdae6208b2d87975a8620ee5"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.492476 7307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:16:37.493389 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.493650 7307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:37.493770 7307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "39cfebd5efe84366873779b8a05b58b8"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "1a15f8b3d18ff0d2fe993cd1fdf5aa15"
server_key_iv: "fb7f439fcdae6208b2d87975a8620ee5"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.493877 7307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:37.497823 7307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:37.498152 7307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:37.498323 7307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:37.503388 7307 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:42719
I20260811 11:16:37.503427 7380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:42719 every 8 connection(s)
I20260811 11:16:37.503839 7307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:16:37.504619 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:37.507831 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8: Bootstrap starting.
I20260811 11:16:37.508551 7381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:37.508644 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 7307
I20260811 11:16:37.508795 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:16:37.509020 7381 log.cc:824] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:37.509054 28772 external_mini_cluster.cc:1428] Setting key 303fd299fba5daf8d4b316fbd7df803f
I20260811 11:16:37.510202 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8: No bootstrap required, opened a new log
I20260811 11:16:37.512286 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.509679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.28.25.62:42719)
0811 11:16:37.509895 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:37.509906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:37.509943 (+ 37us) server_negotiation.cc:407] Connection header received
0811 11:16:37.510629 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:37.510639 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:37.510706 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:37.510887 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:37.511659 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:37.511662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:37.511663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:16:37.511668 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:37.511725 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:16:37.511740 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:37.511842 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:37.511991 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 11:16:37.512061 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260811 11:16:37.512564 7381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cfebd5efe84366873779b8a05b58b8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42719 } }
I20260811 11:16:37.512777 7381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:37.512817 7381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39cfebd5efe84366873779b8a05b58b8, State: Initialized, Role: FOLLOWER
I20260811 11:16:37.512948 7381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cfebd5efe84366873779b8a05b58b8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42719 } }
I20260811 11:16:37.513042 7381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:16:37.513096 7381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:16:37.513154 7381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:37.514149 7381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cfebd5efe84366873779b8a05b58b8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42719 } }
I20260811 11:16:37.514254 7381 leader_election.cc:302] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39cfebd5efe84366873779b8a05b58b8; no voters:
I20260811 11:16:37.514520 7381 leader_election.cc:288] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:16:37.514593 7386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:37.514950 7386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [term 1 LEADER]: Becoming Leader. State: Replica: 39cfebd5efe84366873779b8a05b58b8, State: Running, Role: LEADER
I20260811 11:16:37.515064 7381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:16:37.515122 7386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cfebd5efe84366873779b8a05b58b8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42719 } }
I20260811 11:16:37.515638 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39cfebd5efe84366873779b8a05b58b8. Latest consensus state: current_term: 1 leader_uuid: "39cfebd5efe84366873779b8a05b58b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cfebd5efe84366873779b8a05b58b8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42719 } } }
I20260811 11:16:37.515813 7387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39cfebd5efe84366873779b8a05b58b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cfebd5efe84366873779b8a05b58b8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42719 } } }
I20260811 11:16:37.515910 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:37.515938 7387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:37.516765 7395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:16:37.517503 7395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:16:37.519438 7395 catalog_manager.cc:1347] Generated new cluster ID: d601b44ba7a4438b9dbcab93f32c186d
I20260811 11:16:37.519675 7395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:16:37.533236 7395 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:16:37.533532 7395 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:16:37.538836 7395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39cfebd5efe84366873779b8a05b58b8: Generated new TSK 0
I20260811 11:16:37.539139 7395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:16:37.542124 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42719
--builtin_ntp_servers=127.28.25.20:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 11:16:37.633523 7405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:37.633759 7405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:37.633786 7405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:37.635191 7405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:37.635246 7405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:37.635277 7405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:16:37.636801 7405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:37.637322 7405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:37.637777 7405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:37.644366 7415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:37.644366 7413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:37.644644 7405 server_base.cc:1034] running on GCE node
W20260811 11:16:37.644374 7412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:37.644989 7405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:37.645313 7405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:37.646504 7405 hybrid_clock.cc:648] HybridClock initialized: now 1786446997646487 us; error 33 us; skew 500 ppm
I20260811 11:16:37.648269 7405 webserver.cc:460] Webserver started at http://127.28.25.1:44861/ using document root <none> and password file <none>
I20260811 11:16:37.648561 7405 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:37.648610 7405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:37.648773 7405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:37.649986 7405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "ce604077d59a45ac871ac0ebcb944d5a"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "be48f62e01513c5a50bd0ec9a830fc3c"
server_key_iv: "64f22036fd0847a1d6083ff139856920"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.650374 7405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "ce604077d59a45ac871ac0ebcb944d5a"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "be48f62e01513c5a50bd0ec9a830fc3c"
server_key_iv: "64f22036fd0847a1d6083ff139856920"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.651923 7405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:16:37.652765 7421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.652927 7405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:16:37.653039 7405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "ce604077d59a45ac871ac0ebcb944d5a"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "be48f62e01513c5a50bd0ec9a830fc3c"
server_key_iv: "64f22036fd0847a1d6083ff139856920"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.653118 7405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:37.656741 7405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:37.657063 7405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:37.657215 7405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:37.657444 7405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:37.657800 7405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:37.657933 7405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.657986 7405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:37.658015 7405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.664484 7405 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:42787
I20260811 11:16:37.664526 7542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:42787 every 8 connection(s)
I20260811 11:16:37.664904 7405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:16:37.668212 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 7405
I20260811 11:16:37.668335 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:16:37.668344 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.665766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:54343 (local address 127.28.25.62:42719)
0811 11:16:37.665887 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:37.665890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:37.666091 (+ 201us) server_negotiation.cc:407] Connection header received
0811 11:16:37.666663 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:37.666665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:37.666673 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:37.666726 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:37.668037 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:37.668038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:37.668039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:16:37.668044 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:37.668061 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:16:37.668066 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:37.668148 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:37.668246 (+ 98us) server_negotiation.cc:299] Negotiation successful
0811 11:16:37.668262 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 11:16:37.668565 28772 external_mini_cluster.cc:1428] Setting key 9462dc042b7b16707a9724e3821ad616
I20260811 11:16:37.668569 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.665686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42719 (local address 127.28.25.1:54343)
0811 11:16:37.665953 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:37.665991 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:37.666529 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:37.666803 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:37.666807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:37.666815 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:37.667789 (+ 974us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:16:37.667789 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:37.667819 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:16:37.667916 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:37.668145 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:37.668154 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:37.668157 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:16:37.668258 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 11:16:37.668327 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
I20260811 11:16:37.669289 7543 heartbeater.cc:347] Connected to a master server at 127.28.25.62:42719
I20260811 11:16:37.669453 7543 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:37.669626 7543 heartbeater.cc:511] Master 127.28.25.62:42719 requested a full tablet report, sending...
I20260811 11:16:37.669685 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42719
--builtin_ntp_servers=127.28.25.20:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 11:16:37.670197 7341 ts_manager.cc:194] Registered new tserver with Master: ce604077d59a45ac871ac0ebcb944d5a (127.28.25.1:42787)
W20260811 11:16:37.761250 7546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:37.761513 7546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:37.761555 7546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:37.762825 7546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:37.762899 7546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:37.762945 7546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:16:37.764446 7546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:37.764763 7546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:37.765106 7546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:37.770761 7551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:37.770836 7555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:37.770907 7553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:37.771101 7546 server_base.cc:1034] running on GCE node
I20260811 11:16:37.771603 7546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:37.771906 7546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:37.773181 7546 hybrid_clock.cc:648] HybridClock initialized: now 1786446997773153 us; error 43 us; skew 500 ppm
I20260811 11:16:37.774945 7546 webserver.cc:460] Webserver started at http://127.28.25.2:39893/ using document root <none> and password file <none>
I20260811 11:16:37.775269 7546 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:37.775343 7546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:37.775516 7546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:37.776706 7546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "03c864b3b2c142a492dfa1ad65e0e550"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "e6a5023332ea46c1dd0b83626847e192"
server_key_iv: "b2f1f2af9c531e0cc0b76fe63251d200"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.777120 7546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "03c864b3b2c142a492dfa1ad65e0e550"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "e6a5023332ea46c1dd0b83626847e192"
server_key_iv: "b2f1f2af9c531e0cc0b76fe63251d200"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.778697 7546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:16:37.779647 7561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.779883 7546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:16:37.779991 7546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "03c864b3b2c142a492dfa1ad65e0e550"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "e6a5023332ea46c1dd0b83626847e192"
server_key_iv: "b2f1f2af9c531e0cc0b76fe63251d200"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.780077 7546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:37.783649 7546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:37.783949 7546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:37.784097 7546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:37.784315 7546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:37.784637 7546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:37.784691 7546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.784734 7546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:37.784768 7546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.791553 7546 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:39463
I20260811 11:16:37.791616 7682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:39463 every 8 connection(s)
I20260811 11:16:37.792068 7546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:16:37.795588 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.792940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:46811 (local address 127.28.25.62:42719)
0811 11:16:37.793031 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:37.793033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:37.793298 (+ 265us) server_negotiation.cc:407] Connection header received
0811 11:16:37.793900 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:37.793902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:37.793908 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:37.793974 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:37.795313 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:37.795315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:37.795316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:16:37.795320 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:37.795344 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:16:37.795355 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:37.795425 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:37.795508 (+ 83us) server_negotiation.cc:299] Negotiation successful
0811 11:16:37.795524 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:16:37.795820 7685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.792879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42719 (local address 127.28.25.2:46811)
0811 11:16:37.793112 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:37.793149 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:37.793720 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:37.794062 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:37.794066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:37.794074 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:37.795147 (+ 1073us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:16:37.795147 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:37.795179 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:16:37.795212 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:37.795432 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:37.795437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:37.795440 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:16:37.795502 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 11:16:37.795558 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":123,"threads_started":1}
I20260811 11:16:37.796366 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 7546
I20260811 11:16:37.796504 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:16:37.796716 28772 external_mini_cluster.cc:1428] Setting key cc8f281918c06cebf721a948426dcbb8
I20260811 11:16:37.797251 7683 heartbeater.cc:347] Connected to a master server at 127.28.25.62:42719
I20260811 11:16:37.797379 7683 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:37.797639 7683 heartbeater.cc:511] Master 127.28.25.62:42719 requested a full tablet report, sending...
I20260811 11:16:37.797967 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42719
--builtin_ntp_servers=127.28.25.20:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 11:16:37.798118 7341 ts_manager.cc:194] Registered new tserver with Master: 03c864b3b2c142a492dfa1ad65e0e550 (127.28.25.2:39463)
W20260811 11:16:37.885880 7686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:37.886119 7686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:37.886147 7686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:37.887481 7686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:37.887537 7686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:37.887566 7686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:16:37.889190 7686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:37.889506 7686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:37.889868 7686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:37.895460 7693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:37.895459 7692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:37.895656 7686 server_base.cc:1034] running on GCE node
W20260811 11:16:37.895533 7695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:37.896071 7686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:37.896319 7686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:37.897549 7686 hybrid_clock.cc:648] HybridClock initialized: now 1786446997897518 us; error 51 us; skew 500 ppm
I20260811 11:16:37.904310 7686 webserver.cc:460] Webserver started at http://127.28.25.3:37383/ using document root <none> and password file <none>
I20260811 11:16:37.904637 7686 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:37.904696 7686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:37.904886 7686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:37.906100 7686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "1fd2e653cbae4a01ae908822debc401a"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "16587cb49cb4e872da6bc5c150d23364"
server_key_iv: "2fed03656d0946d6cd1e3d8668d04e02"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.906507 7686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "1fd2e653cbae4a01ae908822debc401a"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "16587cb49cb4e872da6bc5c150d23364"
server_key_iv: "2fed03656d0946d6cd1e3d8668d04e02"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.908191 7686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:16:37.909294 7703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.909569 7686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:16:37.909704 7686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "1fd2e653cbae4a01ae908822debc401a"
format_stamp: "Formatted at 2026-08-11 11:16:37 on dist-test-slave-2w3w"
server_key: "16587cb49cb4e872da6bc5c150d23364"
server_key_iv: "2fed03656d0946d6cd1e3d8668d04e02"
server_key_version: "encryptionkey@0"
I20260811 11:16:37.909796 7686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:37.913622 7686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:37.913966 7686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:37.914130 7686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:37.914422 7686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:37.914793 7686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:37.914853 7686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.914896 7686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:37.914923 7686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:37.921478 7686 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:37729
I20260811 11:16:37.921574 7823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:37729 every 8 connection(s)
I20260811 11:16:37.921944 7686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:16:37.924252 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 7686
I20260811 11:16:37.924347 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:16:37.924607 28772 external_mini_cluster.cc:1428] Setting key 3c72569eb69ec258f041efeb7af8194e
I20260811 11:16:37.925648 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.922808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:40773 (local address 127.28.25.62:42719)
0811 11:16:37.922917 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:37.922920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:37.923180 (+ 260us) server_negotiation.cc:407] Connection header received
0811 11:16:37.923765 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:37.923766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:37.923774 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:37.923819 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:37.925337 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:37.925339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:37.925339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:16:37.925343 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:37.925359 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:16:37.925363 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:37.925442 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:37.925558 (+ 116us) server_negotiation.cc:299] Negotiation successful
0811 11:16:37.925576 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 11:16:37.925889 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:37.922784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42719 (local address 127.28.25.3:40773)
0811 11:16:37.923036 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:37.923084 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:37.923647 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:37.923898 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:37.923903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:37.923912 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:37.925155 (+ 1243us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:16:37.925155 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:37.925190 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:16:37.925223 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:37.925452 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:37.925457 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:37.925461 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:16:37.925559 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 11:16:37.925634 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260811 11:16:37.926400 7824 heartbeater.cc:347] Connected to a master server at 127.28.25.62:42719
I20260811 11:16:37.926501 7824 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:37.926625 7824 heartbeater.cc:511] Master 127.28.25.62:42719 requested a full tablet report, sending...
I20260811 11:16:37.927024 7341 ts_manager.cc:194] Registered new tserver with Master: 1fd2e653cbae4a01ae908822debc401a (127.28.25.3:37729)
I20260811 11:16:37.936878 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 11:16:37.938863 7385 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:16:37.938329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.28.25.62:42719)
0811 11:16:37.938404 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:37.938406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:37.938545 (+ 139us) server_negotiation.cc:407] Connection header received
0811 11:16:37.938587 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:37.938587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:37.938592 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:37.938647 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:37.938783 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39144: BlockingRecv error: recv got EOF from 127.0.0.1:39144 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:16:37.939033 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 7405
I20260811 11:16:37.950080 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 7546
I20260811 11:16:37.957651 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 7686
I20260811 11:16:37.966018 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 7307
2026-08-11T11:16:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (615 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:16:38 dist-test-slave-2w3w krb5kdc[7835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:16:38 dist-test-slave-2w3w krb5kdc[7835](info): set up 2 sockets
Aug 11 11:16:38 dist-test-slave-2w3w krb5kdc[7835](info): commencing operation
krb5kdc: starting...
W20260811 11:16:39.101497 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:16:39 dist-test-slave-2w3w krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:16:39.242923 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:42133
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:42133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:39.329968 7850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:39.330279 7850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:39.330318 7850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:39.331632 7850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:16:39.331717 7850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:39.331741 7850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:39.331763 7850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:16:39.331795 7850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:16:39.333459 7850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:42133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:42133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:39.333791 7850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:39.334060 7850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:39.339654 7857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:39.339821 7850 server_base.cc:1034] running on GCE node
W20260811 11:16:39.339659 7856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:39.339654 7859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:39.340260 7850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:39.340555 7850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:39.341763 7850 hybrid_clock.cc:648] HybridClock initialized: now 1786446999341735 us; error 50 us; skew 500 ppm
Aug 11 11:16:39 dist-test-slave-2w3w krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:39.344741 7850 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:16:39.345337 7850 webserver.cc:460] Webserver started at http://127.28.25.62:40139/ using document root <none> and password file <none>
I20260811 11:16:39.345587 7850 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:39.345657 7850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:39.345897 7850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:39.347148 7850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "178100555ef94c4882a0d8a5994611e9"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "46569e055ddc09a31627abe723895d50"
server_key_iv: "cd64cea4b60fcf022e3b54dd0975ad42"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.347543 7850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "178100555ef94c4882a0d8a5994611e9"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "46569e055ddc09a31627abe723895d50"
server_key_iv: "cd64cea4b60fcf022e3b54dd0975ad42"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.349066 7850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:39.349931 7867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:39.350196 7850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:39.350327 7850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "178100555ef94c4882a0d8a5994611e9"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "46569e055ddc09a31627abe723895d50"
server_key_iv: "cd64cea4b60fcf022e3b54dd0975ad42"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.350430 7850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:39.374770 7850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:39.377367 7850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:39.377578 7850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:39.382287 7850 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:42133
I20260811 11:16:39.382337 7922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:42133 every 8 connection(s)
I20260811 11:16:39.382773 7850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:16:39.383880 7923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:39.386310 7923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9: Bootstrap starting.
I20260811 11:16:39.386914 7923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:39.387179 7923 log.cc:824] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:39.387992 7923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9: No bootstrap required, opened a new log
I20260811 11:16:39.389570 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 7850
I20260811 11:16:39.389685 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:16:39.389886 28772 external_mini_cluster.cc:1428] Setting key 6c7cb42f77f623893c0d81cd09a3777a
I20260811 11:16:39.390129 7923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178100555ef94c4882a0d8a5994611e9" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42133 } }
I20260811 11:16:39.390295 7923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:39.390319 7923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178100555ef94c4882a0d8a5994611e9, State: Initialized, Role: FOLLOWER
I20260811 11:16:39.390695 7923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178100555ef94c4882a0d8a5994611e9" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42133 } }
I20260811 11:16:39.390803 7923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:16:39.390856 7923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:16:39.390933 7923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:39.391599 7923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178100555ef94c4882a0d8a5994611e9" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42133 } }
I20260811 11:16:39.391728 7923 leader_election.cc:302] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 178100555ef94c4882a0d8a5994611e9; no voters:
I20260811 11:16:39.391932 7923 leader_election.cc:288] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:16:39.391970 7928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:39.392120 7928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [term 1 LEADER]: Becoming Leader. State: Replica: 178100555ef94c4882a0d8a5994611e9, State: Running, Role: LEADER
I20260811 11:16:39.392217 7923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:16:39.392218 7928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178100555ef94c4882a0d8a5994611e9" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42133 } }
I20260811 11:16:39.393889 7929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "178100555ef94c4882a0d8a5994611e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178100555ef94c4882a0d8a5994611e9" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42133 } } }
I20260811 11:16:39.394001 7929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [sys.catalog]: This master's current role is: LEADER
I20260811 11:16:39.394361 7930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 178100555ef94c4882a0d8a5994611e9. Latest consensus state: current_term: 1 leader_uuid: "178100555ef94c4882a0d8a5994611e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178100555ef94c4882a0d8a5994611e9" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 42133 } } }
I20260811 11:16:39.394459 7930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9 [sys.catalog]: This master's current role is: LEADER
W20260811 11:16:39.394706 7944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:16:39.394800 7944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:16:39.394862 7945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:16:39.395061 7945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:16:39.396679 7945 catalog_manager.cc:1347] Generated new cluster ID: 95a2f18e15ea48679fede4ece016699f
I20260811 11:16:39.396732 7945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 11:16:39 dist-test-slave-2w3w krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:16:39.402747 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:39.390550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.28.25.62:42133)
0811 11:16:39.390732 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:39.390743 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:39.390800 (+ 57us) server_negotiation.cc:407] Connection header received
0811 11:16:39.390941 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:39.390950 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:39.391047 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:39.391211 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:39.392099 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.393543 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.394879 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.395527 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.395608 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:39.398655 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:39.398660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:39.398661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:39.398674 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:39.400896 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:39.401320 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:39.401324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:39.401324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:39.401454 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:39.401675 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:39.401676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:39.401676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:39.402052 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:39.402275 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:39.402361 (+ 86us) server_negotiation.cc:299] Negotiation successful
0811 11:16:39.402530 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260811 11:16:39.414155 7945 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:16:39.415053 7945 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:16:39.422334 7945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9: Generated new TSK 0
I20260811 11:16:39.422601 7945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:16:39.458797 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42133
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:39.546542 7954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:39.546784 7954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:39.546823 7954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:39.548076 7954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:39.548125 7954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:39.548153 7954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:16:39.549640 7954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:39.549847 7954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:39.550046 7954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:16:39.555917 7954 server_base.cc:1034] running on GCE node
W20260811 11:16:39.555876 7963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:39.555881 7961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:39.555881 7960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:39.556388 7954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:39.556669 7954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:39.557859 7954 hybrid_clock.cc:648] HybridClock initialized: now 1786446999557843 us; error 30 us; skew 500 ppm
Aug 11 11:16:39 dist-test-slave-2w3w krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:39.560931 7954 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:16:39.561458 7954 webserver.cc:460] Webserver started at http://127.28.25.1:41087/ using document root <none> and password file <none>
I20260811 11:16:39.561717 7954 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:39.561801 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:39.561964 7954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:39.563213 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "8514b222874844339a407e81783f4a24"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "0826061031f291c35c121ceb25b4e2d2"
server_key_iv: "652cdd37e5539d4100a1e36f3a0887d6"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.563624 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "8514b222874844339a407e81783f4a24"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "0826061031f291c35c121ceb25b4e2d2"
server_key_iv: "652cdd37e5539d4100a1e36f3a0887d6"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.565176 7954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:16:39.566102 7970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:39.566318 7954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:39.566438 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "8514b222874844339a407e81783f4a24"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "0826061031f291c35c121ceb25b4e2d2"
server_key_iv: "652cdd37e5539d4100a1e36f3a0887d6"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.566522 7954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:39.585189 7954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:39.587956 7954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:39.588177 7954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:39.588462 7954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:39.588835 7954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:39.588892 7954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:39.588927 7954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:39.588955 7954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:39.595852 7954 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:36863
I20260811 11:16:39.595902 8094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:36863 every 8 connection(s)
I20260811 11:16:39.596261 7954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
Aug 11 11:16:39 dist-test-slave-2w3w krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:16:39.605537 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 7954
I20260811 11:16:39.605659 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:16:39.605878 28772 external_mini_cluster.cc:1428] Setting key 220c2c3a1bd8bbe9763836c10f9ec8f8
I20260811 11:16:39.607322 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:39.597020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:41469 (local address 127.28.25.62:42133)
0811 11:16:39.597067 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:39.597069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:39.597336 (+ 267us) server_negotiation.cc:407] Connection header received
0811 11:16:39.598379 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:39.598380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:39.598396 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:39.598474 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:39.599193 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.599805 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.600882 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.601119 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.601185 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:39.603875 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:39.603877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:39.603877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:39.603886 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:39.605274 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:39.606018 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:39.606020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:39.606020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:39.606127 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:39.606456 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:39.606457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:39.606457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:39.606603 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:39.606694 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:39.607171 (+ 477us) server_negotiation.cc:299] Negotiation successful
0811 11:16:39.607254 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 11:16:39.607648 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:39.596997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42133 (local address 127.28.25.1:41469)
0811 11:16:39.597172 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:39.597228 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:39.598221 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:39.598558 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:39.598562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:39.598844 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:39.599090 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:39.599095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.599907 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.599908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:39.600787 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:39.600789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.601210 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.601211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:39.601323 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:39.601967 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:39.601977 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:39.603766 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:39.605466 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:39.605477 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:39.605480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:39.605837 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:39.606231 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:39.606232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:39.606232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:39.606382 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:39.606705 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:39.606713 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:39.606990 (+ 277us) client_negotiation.cc:769] Sending connection context
0811 11:16:39.607121 (+ 131us) client_negotiation.cc:240] Negotiation successful
0811 11:16:39.607318 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260811 11:16:39.608287 8095 heartbeater.cc:347] Connected to a master server at 127.28.25.62:42133
I20260811 11:16:39.608407 8095 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:39.608613 8095 heartbeater.cc:511] Master 127.28.25.62:42133 requested a full tablet report, sending...
I20260811 11:16:39.609189 7886 ts_manager.cc:194] Registered new tserver with Master: 8514b222874844339a407e81783f4a24 (127.28.25.1:36863)
I20260811 11:16:39.609827 7886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:41469
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:16:39.640174 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42133
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:39.730557 8100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:39.730809 8100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:39.730839 8100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:39.732223 8100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:39.732306 8100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:39.732339 8100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:16:39.734011 8100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:39.734311 8100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:39.734591 8100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:39.740321 8106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:39.740370 8100 server_base.cc:1034] running on GCE node
W20260811 11:16:39.740329 8107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:39.740592 8110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:39.740860 8100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:39.741137 8100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:39.742340 8100 hybrid_clock.cc:648] HybridClock initialized: now 1786446999742327 us; error 37 us; skew 500 ppm
Aug 11 11:16:39 dist-test-slave-2w3w krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:39.745191 8100 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:16:39.745784 8100 webserver.cc:460] Webserver started at http://127.28.25.2:42449/ using document root <none> and password file <none>
I20260811 11:16:39.746055 8100 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:39.746135 8100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:39.746335 8100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:39.747586 8100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "8a6ff4ce7a204e549f80b1643da97a7c"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "1202093d285cd3f501fc819c896f8637"
server_key_iv: "bd6376bff1530fd4ca49e0086be1fe8a"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.747989 8100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "8a6ff4ce7a204e549f80b1643da97a7c"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "1202093d285cd3f501fc819c896f8637"
server_key_iv: "bd6376bff1530fd4ca49e0086be1fe8a"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.749495 8100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:16:39.750341 8118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:39.750789 8100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:39.750926 8100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "8a6ff4ce7a204e549f80b1643da97a7c"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "1202093d285cd3f501fc819c896f8637"
server_key_iv: "bd6376bff1530fd4ca49e0086be1fe8a"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.751091 8100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:39.787213 8100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:39.789897 8100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:39.790041 8100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:39.790243 8100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:39.790607 8100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:39.790668 8100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:39.790712 8100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:39.790740 8100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:39.797449 8100 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:37531
I20260811 11:16:39.797546 8243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:37531 every 8 connection(s)
I20260811 11:16:39.797904 8100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
Aug 11 11:16:39 dist-test-slave-2w3w krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:16:39.807514 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 8100
I20260811 11:16:39.807632 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:16:39.807955 28772 external_mini_cluster.cc:1428] Setting key 382823170276f9df2bd6abb6a345ac1d
I20260811 11:16:39.808718 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:39.798720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:60439 (local address 127.28.25.62:42133)
0811 11:16:39.798857 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:39.798859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:39.799113 (+ 254us) server_negotiation.cc:407] Connection header received
0811 11:16:39.799729 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:39.799731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:39.799745 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:39.799813 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:39.800878 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.801387 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.802437 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.802641 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.802728 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:39.805273 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:39.805275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:39.805275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:39.805284 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:39.806813 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:39.807341 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:39.807343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:39.807343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:39.807449 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:39.807848 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:39.807849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:39.807850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:39.808007 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:39.808085 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:39.808516 (+ 431us) server_negotiation.cc:299] Negotiation successful
0811 11:16:39.808619 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 11:16:39.808878 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:39.798699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42133 (local address 127.28.25.2:60439)
0811 11:16:39.798914 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:39.799012 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:39.799601 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:39.799890 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:39.799894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:39.800529 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:39.800750 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:39.800756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.801494 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.801495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:39.802339 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:39.802341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.802745 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.802746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:39.802884 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:39.803508 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:39.803517 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:39.805137 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:39.806912 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:39.806922 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:39.806924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:39.807250 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:39.807562 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:39.807563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:39.807563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:39.807758 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:39.808123 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:39.808132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:39.808330 (+ 198us) client_negotiation.cc:769] Sending connection context
0811 11:16:39.808449 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 11:16:39.808595 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":111,"threads_started":1}
I20260811 11:16:39.809407 8245 heartbeater.cc:347] Connected to a master server at 127.28.25.62:42133
I20260811 11:16:39.809530 8245 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:39.809725 8245 heartbeater.cc:511] Master 127.28.25.62:42133 requested a full tablet report, sending...
I20260811 11:16:39.810155 7886 ts_manager.cc:194] Registered new tserver with Master: 8a6ff4ce7a204e549f80b1643da97a7c (127.28.25.2:37531)
I20260811 11:16:39.810695 7886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:60439
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:16:39.843520 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42133
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:16:39.934465 8250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:16:39.934700 8250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:16:39.934747 8250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:16:39.936055 8250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:16:39.936113 8250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:16:39.936144 8250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:16:39.937727 8250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:42133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:16:39.938009 8250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:16:39.938259 8250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:16:39.944125 8256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:39.944125 8255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:16:39.944129 8258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:16:39.944494 8250 server_base.cc:1034] running on GCE node
I20260811 11:16:39.944729 8250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:16:39.945032 8250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:16:39.946223 8250 hybrid_clock.cc:648] HybridClock initialized: now 1786446999946191 us; error 49 us; skew 500 ppm
Aug 11 11:16:39 dist-test-slave-2w3w krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:16:39.949066 8250 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:16:39.949530 8250 webserver.cc:460] Webserver started at http://127.28.25.3:35233/ using document root <none> and password file <none>
I20260811 11:16:39.949775 8250 fs_manager.cc:359] Metadata directory not provided
I20260811 11:16:39.949823 8250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:16:39.949965 8250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:16:39.951188 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "478c22efcd2e4eca9a76393a193c6b0f"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "64447d06ac3309374da7ad08122f52cc"
server_key_iv: "af3009e87d22a67685c235d78c6d1294"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.951587 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "478c22efcd2e4eca9a76393a193c6b0f"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "64447d06ac3309374da7ad08122f52cc"
server_key_iv: "af3009e87d22a67685c235d78c6d1294"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.953157 8250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:39.954216 8266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:39.954476 8250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:16:39.954602 8250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "478c22efcd2e4eca9a76393a193c6b0f"
format_stamp: "Formatted at 2026-08-11 11:16:39 on dist-test-slave-2w3w"
server_key: "64447d06ac3309374da7ad08122f52cc"
server_key_iv: "af3009e87d22a67685c235d78c6d1294"
server_key_version: "encryptionkey@0"
I20260811 11:16:39.954739 8250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:16:39.963608 8250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:16:39.966212 8250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:16:39.966444 8250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:16:39.966735 8250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:16:39.967168 8250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:16:39.967227 8250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:39.967270 8250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:16:39.967306 8250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:16:39.974531 8250 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:41905
I20260811 11:16:39.974627 8386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:41905 every 8 connection(s)
I20260811 11:16:39.975045 8250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:16:39.980343 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 8250
I20260811 11:16:39.980479 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:16:39.980713 28772 external_mini_cluster.cc:1428] Setting key 4e6e572c8619231d678d8722380578e6
Aug 11 11:16:39 dist-test-slave-2w3w krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786446999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:16:39.986555 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:39.976021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:59175 (local address 127.28.25.62:42133)
0811 11:16:39.976100 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:39.976103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:39.976393 (+ 290us) server_negotiation.cc:407] Connection header received
0811 11:16:39.976915 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:39.976917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:39.976935 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:39.976998 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:39.978114 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.978617 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.979854 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.980085 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.980155 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:39.983340 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:39.983341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:39.983342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:39.983351 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:39.984739 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:39.985295 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:39.985297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:39.985297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:39.985366 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:39.985729 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:39.985729 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:39.985730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:39.985922 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:39.985988 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:39.986381 (+ 393us) server_negotiation.cc:299] Negotiation successful
0811 11:16:39.986473 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:16:39.986728 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:39.975992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:42133 (local address 127.28.25.3:59175)
0811 11:16:39.976193 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:39.976235 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:39.976790 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:39.977061 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:39.977066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:39.977695 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:16:39.977997 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:39.978004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.978730 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.978731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:39.979727 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:39.979730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.980208 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.980209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:39.980432 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:39.981199 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:16:39.981212 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:16:39.983198 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:16:39.984895 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:39.984904 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:39.984907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:39.985165 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:39.985478 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:16:39.985480 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:16:39.985480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:16:39.985644 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:16:39.986024 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:16:39.986032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:16:39.986188 (+ 156us) client_negotiation.cc:769] Sending connection context
0811 11:16:39.986296 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 11:16:39.986495 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":109,"threads_started":1}
I20260811 11:16:39.987339 8387 heartbeater.cc:347] Connected to a master server at 127.28.25.62:42133
I20260811 11:16:39.987470 8387 heartbeater.cc:464] Registering TS with master...
I20260811 11:16:39.987692 8387 heartbeater.cc:511] Master 127.28.25.62:42133 requested a full tablet report, sending...
I20260811 11:16:39.988152 7886 ts_manager.cc:194] Registered new tserver with Master: 478c22efcd2e4eca9a76393a193c6b0f (127.28.25.3:41905)
I20260811 11:16:39.988551 7886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:59175
I20260811 11:16:39.992946 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:16:40.002535 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:39.994574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.28.25.62:42133)
0811 11:16:39.994648 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:39.994651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:39.994839 (+ 188us) server_negotiation.cc:407] Connection header received
0811 11:16:39.994888 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:39.994889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:39.994899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:39.994950 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:16:39.995637 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.996154 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.997109 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:39.997469 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:39.997553 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:39.999228 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:16:39.999230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:16:39.999230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:16:39.999240 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:16:40.001015 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:40.001479 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:40.001480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:40.001481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:40.001547 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:16:40.001748 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:16:40.001748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:16:40.001749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:16:40.001967 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:16:40.002050 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:40.002250 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 11:16:40.002326 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:16:40.004999 7886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:16:40.006973 7886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:16:40.015640 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.011072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.28.25.1:36863)
0811 11:16:40.011437 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:40.011441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:40.011451 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:16:40.011490 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:40.011492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:40.011508 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:40.011589 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:40.011848 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.013036 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.014090 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.014951 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.015061 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.015131 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:40.015436 (+ 305us) server_negotiation.cc:299] Negotiation successful
0811 11:16:40.015525 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":177,"threads_started":1}
I20260811 11:16:40.015641 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.010586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:37531 (local address 127.0.0.1:33526)
0811 11:16:40.010999 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:40.011011 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:40.011108 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:40.011726 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:40.011727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:40.011728 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:40.011770 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:40.011772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.013166 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.013167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:40.014311 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:40.014313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.015343 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.015347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:40.015424 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.015429 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:16:40.015494 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:16:40.015542 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":31,"thread_start_us":123,"threads_started":1}
I20260811 11:16:40.015704 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.010688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.28.25.2:37531)
0811 11:16:40.010918 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:40.010924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:40.011081 (+ 157us) server_negotiation.cc:407] Connection header received
0811 11:16:40.011338 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:40.011342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:40.011370 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:40.011478 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:40.011904 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.013042 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.014404 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.015101 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.015175 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.015246 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:40.015551 (+ 305us) server_negotiation.cc:299] Negotiation successful
0811 11:16:40.015603 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260811 11:16:40.016259 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.010821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36863 (local address 127.0.0.1:48016)
0811 11:16:40.011009 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:40.011018 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:40.011074 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:40.011603 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:40.011604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:40.011606 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:40.011673 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:40.011675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.013192 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.013193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:40.013990 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:40.013992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.015083 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.015085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:40.015185 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.015201 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 11:16:40.015399 (+ 198us) client_negotiation.cc:240] Negotiation successful
0811 11:16:40.015622 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":25,"thread_start_us":74,"threads_started":1}
I20260811 11:16:40.016606 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.010565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:41905 (local address 127.0.0.1:42766)
0811 11:16:40.010778 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:40.010952 (+ 174us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:40.011052 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:40.011524 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:40.011529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:40.011532 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:40.011618 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:40.011620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.012619 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.012620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:40.014110 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:40.014113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.015702 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.015703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:40.016437 (+ 734us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.016442 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:16:40.016501 (+ 59us) client_negotiation.cc:240] Negotiation successful
0811 11:16:40.016547 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260811 11:16:40.016932 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.010779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.28.25.3:41905)
0811 11:16:40.011222 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:40.011226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:40.011234 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:16:40.011300 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:40.011302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:40.011320 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:40.011410 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:40.011732 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.012510 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.014207 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.014951 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.015061 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.015131 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:40.016773 (+ 1642us) server_negotiation.cc:299] Negotiation successful
0811 11:16:40.016837 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":106,"threads_started":1}
I20260811 11:16:40.016826 8171 tablet_service.cc:1467] Processing CreateTablet for tablet 653f23cf96954ff39850a5186e9fb16c (DEFAULT_TABLE table=test-table [id=111ba4ceea6d48748a6e6988f211547c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:40.017089 8022 tablet_service.cc:1467] Processing CreateTablet for tablet 653f23cf96954ff39850a5186e9fb16c (DEFAULT_TABLE table=test-table [id=111ba4ceea6d48748a6e6988f211547c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:40.017253 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653f23cf96954ff39850a5186e9fb16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:40.017477 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653f23cf96954ff39850a5186e9fb16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:40.017596 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 653f23cf96954ff39850a5186e9fb16c (DEFAULT_TABLE table=test-table [id=111ba4ceea6d48748a6e6988f211547c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:16:40.017985 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653f23cf96954ff39850a5186e9fb16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:16:40.020427 8406 tablet_bootstrap.cc:492] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: Bootstrap starting.
I20260811 11:16:40.021224 8407 tablet_bootstrap.cc:492] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24: Bootstrap starting.
I20260811 11:16:40.021291 8406 tablet_bootstrap.cc:654] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:40.021677 8408 tablet_bootstrap.cc:492] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: Bootstrap starting.
I20260811 11:16:40.022007 8407 tablet_bootstrap.cc:654] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:40.022006 8406 log.cc:824] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:40.022382 8408 tablet_bootstrap.cc:654] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: Neither blocks nor log segments found. Creating new log.
I20260811 11:16:40.022601 8407 log.cc:824] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:40.022950 8406 tablet_bootstrap.cc:492] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: No bootstrap required, opened a new log
I20260811 11:16:40.023005 8408 log.cc:824] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: Log is configured to *not* fsync() on all Append() calls
I20260811 11:16:40.023066 8406 ts_tablet_manager.cc:1397] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:16:40.023481 8407 tablet_bootstrap.cc:492] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24: No bootstrap required, opened a new log
I20260811 11:16:40.023556 8407 ts_tablet_manager.cc:1397] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:16:40.023774 8408 tablet_bootstrap.cc:492] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: No bootstrap required, opened a new log
I20260811 11:16:40.023870 8408 ts_tablet_manager.cc:1397] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:16:40.025256 8406 raft_consensus.cc:348] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 } } peers { permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 } } peers { permanent_uuid: "478c22efcd2e4eca9a76393a193c6b0f" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 41905 } }
I20260811 11:16:40.025480 8406 raft_consensus.cc:374] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:40.025523 8406 raft_consensus.cc:729] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6ff4ce7a204e549f80b1643da97a7c, State: Initialized, Role: FOLLOWER
I20260811 11:16:40.025689 8406 consensus_queue.cc:260] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 } } peers { permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 } } peers { permanent_uuid: "478c22efcd2e4eca9a76393a193c6b0f" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 41905 } }
I20260811 11:16:40.025641 8408 raft_consensus.cc:348] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 } } peers { permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 } } peers { permanent_uuid: "478c22efcd2e4eca9a76393a193c6b0f" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 41905 } }
I20260811 11:16:40.025813 8408 raft_consensus.cc:374] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:40.025856 8408 raft_consensus.cc:729] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478c22efcd2e4eca9a76393a193c6b0f, State: Initialized, Role: FOLLOWER
I20260811 11:16:40.025976 8408 consensus_queue.cc:260] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 } } peers { permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 } } peers { permanent_uuid: "478c22efcd2e4eca9a76393a193c6b0f" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 41905 } }
I20260811 11:16:40.025846 8407 raft_consensus.cc:348] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 } } peers { permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 } } peers { permanent_uuid: "478c22efcd2e4eca9a76393a193c6b0f" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 41905 } }
I20260811 11:16:40.026141 8407 raft_consensus.cc:374] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:16:40.026183 8406 ts_tablet_manager.cc:1428] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:16:40.026225 8407 raft_consensus.cc:729] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8514b222874844339a407e81783f4a24, State: Initialized, Role: FOLLOWER
I20260811 11:16:40.026294 8408 ts_tablet_manager.cc:1428] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 11:16:40.026335 8245 heartbeater.cc:503] Master 127.28.25.62:42133 was elected leader, sending a full tablet report...
I20260811 11:16:40.026389 8407 consensus_queue.cc:260] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 } } peers { permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 } } peers { permanent_uuid: "478c22efcd2e4eca9a76393a193c6b0f" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 41905 } }
I20260811 11:16:40.026703 8387 heartbeater.cc:503] Master 127.28.25.62:42133 was elected leader, sending a full tablet report...
I20260811 11:16:40.026731 8095 heartbeater.cc:503] Master 127.28.25.62:42133 was elected leader, sending a full tablet report...
I20260811 11:16:40.026726 8407 ts_tablet_manager.cc:1428] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260811 11:16:40.048619 8246 tablet.cc:2369] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:16:40.076574 8414 raft_consensus.cc:482] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:16:40.076802 8414 raft_consensus.cc:504] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 } } peers { permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 } } peers { permanent_uuid: "478c22efcd2e4eca9a76393a193c6b0f" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 41905 } }
I20260811 11:16:40.077257 8414 leader_election.cc:288] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a6ff4ce7a204e549f80b1643da97a7c (127.28.25.2:37531), 8514b222874844339a407e81783f4a24 (127.28.25.1:36863)
I20260811 11:16:40.081992 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.077862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:42955 (local address 127.28.25.1:36863)
0811 11:16:40.077906 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:40.077908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:40.078087 (+ 179us) server_negotiation.cc:407] Connection header received
0811 11:16:40.078128 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:40.078129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:40.078140 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:40.078185 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:40.078348 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.079079 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.080833 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.081508 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.081597 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.081609 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:40.081865 (+ 256us) server_negotiation.cc:299] Negotiation successful
0811 11:16:40.081917 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:16:40.081995 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.077852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:54629 (local address 127.28.25.2:37531)
0811 11:16:40.077906 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:40.077908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:40.077919 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:16:40.077953 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:40.077954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:40.077969 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:40.078043 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:16:40.078261 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.079082 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.080829 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.081508 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.081597 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.081609 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:40.081865 (+ 256us) server_negotiation.cc:299] Negotiation successful
0811 11:16:40.081916 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 11:16:40.082001 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.077617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:37531 (local address 127.28.25.3:54629)
0811 11:16:40.077680 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:40.077693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:40.077743 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:40.078062 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:40.078063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:40.078066 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:40.078149 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:40.078156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.079193 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.079194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:40.080651 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:40.080655 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.081621 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.081622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:40.081734 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.081741 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:16:40.081809 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 11:16:40.081863 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":99}
I20260811 11:16:40.082001 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.077747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36863 (local address 127.28.25.3:42955)
0811 11:16:40.077978 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:40.077988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:16:40.078069 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:16:40.078206 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:16:40.078207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:16:40.078208 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:16:40.078268 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:40.078270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.079193 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.079194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:40.080651 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:16:40.080655 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.081621 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.081622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:16:40.081734 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.081742 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 11:16:40.081809 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 11:16:40.081863 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":40,"thread_start_us":181,"threads_started":1}
I20260811 11:16:40.082494 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653f23cf96954ff39850a5186e9fb16c" candidate_uuid: "478c22efcd2e4eca9a76393a193c6b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" is_pre_election: true
I20260811 11:16:40.082504 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653f23cf96954ff39850a5186e9fb16c" candidate_uuid: "478c22efcd2e4eca9a76393a193c6b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8514b222874844339a407e81783f4a24" is_pre_election: true
I20260811 11:16:40.082693 8192 raft_consensus.cc:2454] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 478c22efcd2e4eca9a76393a193c6b0f in term 0.
I20260811 11:16:40.082715 8043 raft_consensus.cc:2454] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 478c22efcd2e4eca9a76393a193c6b0f in term 0.
I20260811 11:16:40.083026 8267 leader_election.cc:302] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 478c22efcd2e4eca9a76393a193c6b0f, 8a6ff4ce7a204e549f80b1643da97a7c; no voters:
I20260811 11:16:40.083233 8414 raft_consensus.cc:2781] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:16:40.083341 8414 raft_consensus.cc:482] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:16:40.083398 8414 raft_consensus.cc:3037] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:40.084193 8414 raft_consensus.cc:504] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 } } peers { permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 } } peers { permanent_uuid: "478c22efcd2e4eca9a76393a193c6b0f" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 41905 } }
I20260811 11:16:40.084323 8414 leader_election.cc:288] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [CANDIDATE]: Term 1 election: Requested vote from peers 8a6ff4ce7a204e549f80b1643da97a7c (127.28.25.2:37531), 8514b222874844339a407e81783f4a24 (127.28.25.1:36863)
I20260811 11:16:40.084570 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653f23cf96954ff39850a5186e9fb16c" candidate_uuid: "478c22efcd2e4eca9a76393a193c6b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a6ff4ce7a204e549f80b1643da97a7c"
I20260811 11:16:40.084580 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653f23cf96954ff39850a5186e9fb16c" candidate_uuid: "478c22efcd2e4eca9a76393a193c6b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8514b222874844339a407e81783f4a24"
I20260811 11:16:40.084698 8192 raft_consensus.cc:3037] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:40.084692 8043 raft_consensus.cc:3037] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:16:40.085585 8043 raft_consensus.cc:2454] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 478c22efcd2e4eca9a76393a193c6b0f in term 1.
I20260811 11:16:40.085747 8192 raft_consensus.cc:2454] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 478c22efcd2e4eca9a76393a193c6b0f in term 1.
I20260811 11:16:40.085793 8267 leader_election.cc:302] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 478c22efcd2e4eca9a76393a193c6b0f, 8514b222874844339a407e81783f4a24; no voters:
I20260811 11:16:40.085919 8414 raft_consensus.cc:2781] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:16:40.086120 8414 raft_consensus.cc:686] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 1 LEADER]: Becoming Leader. State: Replica: 478c22efcd2e4eca9a76393a193c6b0f, State: Running, Role: LEADER
I20260811 11:16:40.086253 8414 consensus_queue.cc:237] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 } } peers { permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 } } peers { permanent_uuid: "478c22efcd2e4eca9a76393a193c6b0f" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 41905 } }
I20260811 11:16:40.087294 7885 catalog_manager.cc:5495] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 478c22efcd2e4eca9a76393a193c6b0f (127.28.25.3). New cstate: current_term: 1 leader_uuid: "478c22efcd2e4eca9a76393a193c6b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "478c22efcd2e4eca9a76393a193c6b0f" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 41905 } health_report { overall_health: HEALTHY } } }
W20260811 11:16:40.097222 8096 tablet.cc:2369] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:16:40.111881 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.109137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.28.25.3:41905)
0811 11:16:40.109211 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 11:16:40.109213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:16:40.109235 (+ 22us) server_negotiation.cc:407] Connection header received
0811 11:16:40.109290 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:16:40.109291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:16:40.109320 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:16:40.109397 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:16:40.109548 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.110097 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.110926 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:16:40.111194 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:16:40.111272 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:16:40.111479 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:16:40.111593 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:16:40.111686 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:16:40.111726 (+ 40us) server_negotiation.cc:299] Negotiation successful
0811 11:16:40.111750 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:16:40.114118 8192 raft_consensus.cc:1261] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c [term 1 FOLLOWER]: Refusing update from remote peer 478c22efcd2e4eca9a76393a193c6b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:40.114209 8043 raft_consensus.cc:1261] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24 [term 1 FOLLOWER]: Refusing update from remote peer 478c22efcd2e4eca9a76393a193c6b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:16:40.114691 8418 consensus_queue.cc:1035] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a6ff4ce7a204e549f80b1643da97a7c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 37531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:40.114869 8414 consensus_queue.cc:1035] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8514b222874844339a407e81783f4a24" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:16:40.118106 8423 mvcc.cc:205] Tried to move back new op lower bound from 7317286912463888384 to 7317286912354217984. Current Snapshot: MvccSnapshot[applied={T|T < 7317286912463888384}]
I20260811 11:16:40.121367 7885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:16:40.121541 7885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:16:40.122623 7885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 178100555ef94c4882a0d8a5994611e9: Sending DeleteTablet for 3 replicas of tablet 653f23cf96954ff39850a5186e9fb16c
I20260811 11:16:40.123111 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet 653f23cf96954ff39850a5186e9fb16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:40 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:42766
I20260811 11:16:40.123234 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet 653f23cf96954ff39850a5186e9fb16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:40 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:33526
I20260811 11:16:40.123286 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet 653f23cf96954ff39850a5186e9fb16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:16:40 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:48016
I20260811 11:16:40.123548 8432 tablet_replica.cc:330] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24: stopping tablet replica
I20260811 11:16:40.123703 8432 raft_consensus.cc:2229] T 653f23cf96954ff39850a5186e9fb16c P 8514b222874844339a407e81783f4a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:16:40.123773 8431 tablet_replica.cc:330] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: stopping tablet replica
I20260811 11:16:40.123903 8431 raft_consensus.cc:2229] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:16:40.123920 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 7954
I20260811 11:16:40.124059 8431 raft_consensus.cc:2258] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:40.128599 8431 ts_tablet_manager.cc:1905] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:40.131425 8431 ts_tablet_manager.cc:1918] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:16:40.131561 8431 log.cc:1196] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/653f23cf96954ff39850a5186e9fb16c
I20260811 11:16:40.131966 8431 ts_tablet_manager.cc:1939] T 653f23cf96954ff39850a5186e9fb16c P 478c22efcd2e4eca9a76393a193c6b0f: Deleting consensus metadata
I20260811 11:16:40.132582 8433 tablet_replica.cc:330] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: stopping tablet replica
I20260811 11:16:40.132656 7872 catalog_manager.cc:4841] TS 478c22efcd2e4eca9a76393a193c6b0f (127.28.25.3:41905): tablet 653f23cf96954ff39850a5186e9fb16c (table test-table [id=111ba4ceea6d48748a6e6988f211547c]) successfully deleted
I20260811 11:16:40.132754 8433 raft_consensus.cc:2229] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:16:40.132946 8433 raft_consensus.cc:2258] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:16:40.133548 8433 ts_tablet_manager.cc:1905] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:16:40.135831 8433 ts_tablet_manager.cc:1918] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:16:40.135953 8433 log.cc:1196] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/653f23cf96954ff39850a5186e9fb16c
I20260811 11:16:40.136250 8433 ts_tablet_manager.cc:1939] T 653f23cf96954ff39850a5186e9fb16c P 8a6ff4ce7a204e549f80b1643da97a7c: Deleting consensus metadata
I20260811 11:16:40.136950 7868 catalog_manager.cc:4841] TS 8a6ff4ce7a204e549f80b1643da97a7c (127.28.25.2:37531): tablet 653f23cf96954ff39850a5186e9fb16c (table test-table [id=111ba4ceea6d48748a6e6988f211547c]) successfully deleted
W20260811 11:16:40.139647 7868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:36863 (error 108)
I20260811 11:16:40.139932 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 8100
I20260811 11:16:40.143390 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:16:40.140720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36863 (local address 127.0.0.1:48026)
0811 11:16:40.143231 (+ 2511us) negotiation.cc:107] Waiting for socket to connect
0811 11:16:40.143319 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:36863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2471}
W20260811 11:16:40.144754 7868 catalog_manager.cc:4584] TS 8514b222874844339a407e81783f4a24 (127.28.25.1:36863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 653f23cf96954ff39850a5186e9fb16c: Network error: Client connection negotiation failed: client connection to 127.28.25.1:36863: connect: Connection refused (error 111)
I20260811 11:16:40.151089 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 8250
I20260811 11:16:40.162822 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 7850
2026-08-11T11:16:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2199 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:16:40 dist-test-slave-2w3w krb5kdc[8437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:16:40 dist-test-slave-2w3w krb5kdc[8437](info): set up 2 sockets
Aug 11 11:16:40 dist-test-slave-2w3w krb5kdc[8437](info): commencing operation
krb5kdc: starting...
W20260811 11:16:41.298703 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:16:41 dist-test-slave-2w3w krb5kdc[8437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:16:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.25.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.25.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/rangeradmin_127.28.25.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/rangeradmin_127.28.25.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.25.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.25.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/rangerlookup_127.28.25.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/rangerlookup_127.28.25.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.25.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.25.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/HTTP_127.28.25.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/HTTP_127.28.25.20@KRBTEST.COM.keytab.
I20260811 11:16:41.507494 28772 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskR10UZg/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/postgres -l logfile start
2026-08-11 11:16:44.626 UTC [8471] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 11:16:44.627 UTC [8471] LOG: listening on IPv4 address "127.28.25.20", port 51231
2026-08-11 11:16:44.632 UTC [8471] LOG: listening on Unix socket "/tmp/.s.PGSQL.51231"
2026-08-11 11:16:44.672 UTC [8474] LOG: database system was shut down at 2026-08-11 11:16:42 UTC
2026-08-11 11:16:44.677 UTC [8471] LOG: database system is ready to accept connections
I20260811 11:16:45.682407 28772 mini_postgres.cc:96] Postgres bound to 51231
2026-08-11 11:16:45.690 UTC [8484] FATAL: database "slave" does not exist
127.28.25.20:51231 - accepting connections
I20260811 11:16:45.691679 28772 mini_ranger.cc:161] Starting Ranger...
I20260811 11:16:45.709852 28772 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 11:16:46.317279 28772 mini_ranger.cc:87] Created ranger Postgres database
I20260811 11:16:46.317361 28772 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-admin
2026-08-11 11:16:46,727 [I] DB FLAVOR :POSTGRES
2026-08-11 11:16:46,727 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 11:16:46,728 [I] Checking connection..
2026-08-11 11:16:46,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 11:16:47,034 [I] Checking connection passed.
2026-08-11 11:16:47,035 [I] --------- Verifying version history table ---------
2026-08-11 11:16:47,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:16:47,321 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 11:16:47,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:16:47,588 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 11:16:47,589 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 11:16:47,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 11:16:47.860 UTC [8553] WARNING: there is no transaction in progress
2026-08-11 11:16:47,869 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 11:16:47,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:16:48,141 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 11:16:48,141 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 11:16:48,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 11:16:48,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 11:16:48,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2w3w.c.gcp-upstream.internal','N') ;"
2026-08-11 11:16:48,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 11:16:49,247 [I] Table x_portal_user does not exist in database ranger
2026-08-11 11:16:49,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 11:16:49,515 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 11:16:49,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 11:16:49,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 11:16:50,084 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 11:16:50,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 11:16:51.307 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.329 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.352 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.373 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.398 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.426 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.500 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.510 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.529 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.542 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.554 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.572 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.584 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.593 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.603 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:51.613 UTC [8672] WARNING: there is no transaction in progress
2026-08-11 11:16:52,578 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 11:16:52,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 11:16:52,879 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 11:16:52,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 11:16:53,164 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 11:16:53,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 11:16:53,464 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 11:16:53,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 11:16:53,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 11:16:54,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2w3w.c.gcp-upstream.internal';"
2026-08-11 11:16:54,275 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 11:16:54,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 11:16:54,552 [I] DB_PATCHES have already been applied
I20260811 11:16:54.559108 28772 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-admin:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskR10UZg/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 11:16:54.559198 28772 mini_ranger.cc:193] Using host: 127.28.25.20
I20260811 11:16:54.563357 28772 mini_ranger.cc:227] Ranger admin URL: http://127.28.25.20:53591
Aug 11, 2026 11:16:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 11:16:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 11:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 11:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 11:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 11:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 11:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 11:16:55.787905 28772 mini_ranger.cc:160] Time spent starting Ranger: real 10.096s user 0.000s sys 0.013s
Aug 11, 2026 11:17:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 11:17:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 11, 2026 11:17:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 11:17:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 11:17:21 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 11:17:21.763569 28772 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.25.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.25.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/rangerkms_127.28.25.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/rangerkms_127.28.25.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.25.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.25.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.25.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/HTTP_127.28.25.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.25.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/HTTP_127.28.25.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 11:17:21 dist-test-slave-2w3w krb5kdc[8437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447041, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 11:17:21.865540 28772 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 11:17:21.878830 28772 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 11:17:23.046598 28772 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 11:17:23.046691 28772 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-kms
I20260811 11:17:23.046713 28772 mini_ranger_kms.cc:227] Using postgres at 127.28.25.20:51231
2026-08-11 11:17:23,256 [I] DB FLAVOR :POSTGRES
2026-08-11 11:17:23,256 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 11:17:23,256 [I] Checking connection
2026-08-11 11:17:23,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 11:17:23,540 [I] connection success
2026-08-11 11:17:23,540 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 11:17:23,540 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 11:17:23,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 11:17:23,810 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 11:17:23,810 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 11:17:23,810 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 11:17:23,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:51231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 11:17:24,099 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 11:17:24.666496 28772 mini_ranger_kms.cc:310] Created kms service
I20260811 11:17:24.910295 28772 mini_ranger_kms.cc:326] Created kudu user
I20260811 11:17:24.963506 28772 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 11:17:24 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447041, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:17:25.429656 28772 mini_ranger_kms.cc:384] Added ranger policy
I20260811 11:17:25.429953 28772 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-kms:/tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 11:17:25.429980 28772 mini_ranger_kms.cc:240] Using host: 127.28.25.20
I20260811 11:17:25.434242 28772 mini_ranger_kms.cc:276] Ranger KMS PID: 8875
I20260811 11:17:25.434394 28772 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.25.20:54507
Aug 11, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 11:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260811 11:17:28.143712 28772 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.278s user 0.000s sys 0.192s
I20260811 11:17:28.143841 28772 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 11:17:28.143920 28772 mini_ranger_kms.cc:401] 127.28.25.20:54507/kms/v1/keys
Aug 11 11:17:28 dist-test-slave-2w3w krb5kdc[8437](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786447048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.25.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 11 11:17:28 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.25.20@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
Aug 11 11:17:29 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.25.20@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
Aug 11 11:17:31 dist-test-slave-2w3w krb5kdc[8437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:17:31.568086 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:46647
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:36903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:46647
--ranger_config_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:54507/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:17:31.679612 8929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:17:31.679870 8929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:17:31.679904 8929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:17:31.681284 8929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:17:31.681354 8929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:17:31.681368 8929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:17:31.681381 8929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:17:31.681396 8929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:17:31.683357 8929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.25.20:54507/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:46647
--ranger_config_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:46647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:17:31.683679 8929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:17:31.683969 8929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:17:31.690164 8929 server_base.cc:1034] running on GCE node
W20260811 11:17:31.690165 8934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:17:31.690403 8938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:17:31.690285 8936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:17:31.690918 8929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:17:31.691313 8929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:17:31.692508 8929 hybrid_clock.cc:648] HybridClock initialized: now 1786447051692479 us; error 50 us; skew 500 ppm
Aug 11 11:17:31 dist-test-slave-2w3w krb5kdc[8437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447051, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:17:31.695751 8929 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:17:31.696426 8929 webserver.cc:460] Webserver started at http://127.28.25.62:46517/ using document root <none> and password file <none>
I20260811 11:17:31.696745 8929 fs_manager.cc:359] Metadata directory not provided
I20260811 11:17:31.696813 8929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:17:31.696998 8929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:17:31 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447051, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:17:31.768319 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "ca32c346dd5442d5ad72064ec0d5594c"
format_stamp: "Formatted at 2026-08-11 11:17:31 on dist-test-slave-2w3w"
server_key: "0473ace2fe28ba3181d341a827efeecb"
server_key_iv: "e48421844c0d6ca20befa95e50917b03"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:31.768796 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "ca32c346dd5442d5ad72064ec0d5594c"
format_stamp: "Formatted at 2026-08-11 11:17:31 on dist-test-slave-2w3w"
server_key: "0473ace2fe28ba3181d341a827efeecb"
server_key_iv: "e48421844c0d6ca20befa95e50917b03"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:31.770462 8929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260811 11:17:31.787427 8947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:17:31.787629 8929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:17:31.787734 8929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "ca32c346dd5442d5ad72064ec0d5594c"
format_stamp: "Formatted at 2026-08-11 11:17:31 on dist-test-slave-2w3w"
server_key: "0473ace2fe28ba3181d341a827efeecb"
server_key_iv: "e48421844c0d6ca20befa95e50917b03"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:31.787846 8929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:17:31.820470 8929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:17:31.822798 8929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:17:31.822974 8929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:17:31.828503 8929 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:46647
I20260811 11:17:31.828531 9008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:46647 every 8 connection(s)
I20260811 11:17:31.829180 8929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:17:31.829974 9009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:17:31.832836 9009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c: Bootstrap starting.
I20260811 11:17:31.833659 9009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c: Neither blocks nor log segments found. Creating new log.
I20260811 11:17:31.833986 9009 log.cc:824] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c: Log is configured to *not* fsync() on all Append() calls
I20260811 11:17:31.834776 9009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c: No bootstrap required, opened a new log
I20260811 11:17:31.837162 9009 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c346dd5442d5ad72064ec0d5594c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46647 } }
I20260811 11:17:31.837376 9009 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:17:31.837457 9009 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca32c346dd5442d5ad72064ec0d5594c, State: Initialized, Role: FOLLOWER
I20260811 11:17:31.837663 9009 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c346dd5442d5ad72064ec0d5594c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46647 } }
I20260811 11:17:31.837771 9009 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:17:31.837817 9009 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:17:31.837869 9009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:17:31.838140 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 8929
I20260811 11:17:31.838258 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:17:31.838490 28772 external_mini_cluster.cc:1428] Setting key 2e5986c8d402901babf96b820dc5c4e1
I20260811 11:17:31.838739 9009 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c346dd5442d5ad72064ec0d5594c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46647 } }
I20260811 11:17:31.838902 9009 leader_election.cc:302] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca32c346dd5442d5ad72064ec0d5594c; no voters:
I20260811 11:17:31.839170 9009 leader_election.cc:288] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:17:31.839449 9013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:17:31.839537 9009 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:17:31.839704 9013 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [term 1 LEADER]: Becoming Leader. State: Replica: ca32c346dd5442d5ad72064ec0d5594c, State: Running, Role: LEADER
I20260811 11:17:31.840448 9013 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c346dd5442d5ad72064ec0d5594c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46647 } }
I20260811 11:17:31.841969 9009 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 11:17:31.842703 9016 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca32c346dd5442d5ad72064ec0d5594c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c346dd5442d5ad72064ec0d5594c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46647 } } }
I20260811 11:17:31.842867 9016 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [sys.catalog]: This master's current role is: LEADER
I20260811 11:17:31.842724 9015 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca32c346dd5442d5ad72064ec0d5594c. Latest consensus state: current_term: 1 leader_uuid: "ca32c346dd5442d5ad72064ec0d5594c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c346dd5442d5ad72064ec0d5594c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46647 } } }
I20260811 11:17:31.843061 9015 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c [sys.catalog]: This master's current role is: LEADER
I20260811 11:17:31.845588 9032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:17:31.847535 9032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 11 11:17:31 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:17:31.854645 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:31.839105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.28.25.62:46647)
0811 11:17:31.839422 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:31.839433 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:31.839474 (+ 41us) server_negotiation.cc:407] Connection header received
0811 11:17:31.839625 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:31.839633 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:31.839698 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:31.839862 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:17:31.840761 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:31.842391 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:31.844785 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:31.846311 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:31.846522 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:31.849859 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:17:31.849865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:17:31.849866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:17:31.849880 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:17:31.852105 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:17:31.852582 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:17:31.852587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:17:31.852587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:17:31.852713 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:17:31.852930 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:17:31.852931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:17:31.852932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:17:31.853337 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:17:31.853484 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:31.853981 (+ 497us) server_negotiation.cc:299] Negotiation successful
0811 11:17:31.854128 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":100,"threads_started":1}
I20260811 11:17:31.858561 9032 catalog_manager.cc:1347] Generated new cluster ID: 2440638474b4468ba5e96edae97121d4
I20260811 11:17:31.858645 9032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:17:31.873543 9032 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:17:31.874501 9032 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:17:31.881021 9032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c: Generated new TSK 0
I20260811 11:17:31.881336 9032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:17:32.700448 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:54507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46647
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:36903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:17:32.824245 9058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:17:32.824565 9058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:17:32.824604 9058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:17:32.826213 9058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:17:32.826290 9058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:17:32.826323 9058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:17:32.828267 9058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.25.20:54507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:17:32.828620 9058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:17:32.828909 9058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:17:32.834925 9069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:17:32.835132 9058 server_base.cc:1034] running on GCE node
W20260811 11:17:32.835207 9066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:17:32.834926 9067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:17:32.835510 9058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:17:32.835843 9058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:17:32.837007 9058 hybrid_clock.cc:648] HybridClock initialized: now 1786447052836994 us; error 29 us; skew 500 ppm
Aug 11 11:17:32 dist-test-slave-2w3w krb5kdc[8437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447052, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:17:32.840396 9058 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:17:32.841003 9058 webserver.cc:460] Webserver started at http://127.28.25.1:38677/ using document root <none> and password file <none>
I20260811 11:17:32.841288 9058 fs_manager.cc:359] Metadata directory not provided
I20260811 11:17:32.841356 9058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:17:32.841521 9058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:17:32 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447052, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:17:32.862084 9058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "12c661f457fc49d38289428d1feff707"
format_stamp: "Formatted at 2026-08-11 11:17:32 on dist-test-slave-2w3w"
server_key: "45f6d132380d029943b51a2713612a8c"
server_key_iv: "4f8bf7559ecb3dc8e0405e86c1c60b29"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:32.862571 9058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "12c661f457fc49d38289428d1feff707"
format_stamp: "Formatted at 2026-08-11 11:17:32 on dist-test-slave-2w3w"
server_key: "45f6d132380d029943b51a2713612a8c"
server_key_iv: "4f8bf7559ecb3dc8e0405e86c1c60b29"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:32.865764 9058 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260811 11:17:32.880306 9079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:17:32.880522 9058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:17:32.880622 9058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "12c661f457fc49d38289428d1feff707"
format_stamp: "Formatted at 2026-08-11 11:17:32 on dist-test-slave-2w3w"
server_key: "45f6d132380d029943b51a2713612a8c"
server_key_iv: "4f8bf7559ecb3dc8e0405e86c1c60b29"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:32.880734 9058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:17:32.897295 9058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:17:32.900143 9058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:17:32.900346 9058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:17:32.900604 9058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:17:32.901046 9058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:17:32.901103 9058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:17:32.901141 9058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:17:32.901168 9058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:17:32.909045 9058 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:40533
I20260811 11:17:32.909327 9204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:40533 every 8 connection(s)
I20260811 11:17:32.909578 9058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
Aug 11 11:17:32 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447052, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:17:32.919741 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 9058
I20260811 11:17:32.919866 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:17:32.920071 28772 external_mini_cluster.cc:1428] Setting key 6fdcfb18122728b3699f300d394b00a6
I20260811 11:17:32.930385 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:32.910671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46647 (local address 127.28.25.1:43795)
0811 11:17:32.910912 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 11:17:32.911008 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0811 11:17:32.911875 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:17:32.912203 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:17:32.912208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:17:32.912506 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:17:32.912801 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:32.912807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:32.913848 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:32.913849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:32.915316 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:32.915321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:32.915927 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:32.915928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:32.916095 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:32.916815 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:17:32.916827 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:17:32.918660 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:17:32.928262 (+ 9602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:17:32.928279 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:17:32.928282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:17:32.928725 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:17:32.929031 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:17:32.929033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:17:32.929033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:17:32.929212 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:17:32.929571 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:17:32.929579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:17:32.929813 (+ 234us) client_negotiation.cc:769] Sending connection context
0811 11:17:32.929930 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 11:17:32.930119 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20260811 11:17:32.931026 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:32.911068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:43795 (local address 127.28.25.62:46647)
0811 11:17:32.911212 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:32.911214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:32.911233 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:17:32.912017 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:32.912019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:32.912035 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:32.912120 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:17:32.912932 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:32.913720 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:32.915467 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:32.915772 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:32.915841 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:32.918807 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:17:32.918809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:17:32.918809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:17:32.918819 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:17:32.928097 (+ 9278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:17:32.928836 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:17:32.928838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:17:32.928838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:17:32.928945 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:17:32.929293 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:17:32.929294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:17:32.929295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:17:32.929473 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:17:32.929540 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:32.930786 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0811 11:17:32.930892 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260811 11:17:32.931845 9205 heartbeater.cc:347] Connected to a master server at 127.28.25.62:46647
I20260811 11:17:32.931953 9205 heartbeater.cc:464] Registering TS with master...
I20260811 11:17:32.932164 9205 heartbeater.cc:511] Master 127.28.25.62:46647 requested a full tablet report, sending...
I20260811 11:17:32.932823 8966 ts_manager.cc:194] Registered new tserver with Master: 12c661f457fc49d38289428d1feff707 (127.28.25.1:40533)
I20260811 11:17:32.933718 8966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:43795
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:17:32.974757 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:54507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46647
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:36903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:17:33.115257 9211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:17:33.115635 9211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:17:33.115676 9211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:17:33.117838 9211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:17:33.117936 9211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:17:33.117985 9211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:17:33.121300 9211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.25.20:54507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:17:33.121636 9211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:17:33.121966 9211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:17:33.129113 9221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:17:33.129125 9220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:17:33.129508 9224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:17:33.130767 9211 server_base.cc:1034] running on GCE node
I20260811 11:17:33.131045 9211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:17:33.131356 9211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Aug 11 11:17:33 dist-test-slave-2w3w krb5kdc[8437](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786447053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:17:33.134662 9211 hybrid_clock.cc:648] HybridClock initialized: now 1786447053134626 us; error 49 us; skew 500 ppm
Aug 11 11:17:33 dist-test-slave-2w3w krb5kdc[8437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:17:33.138425 9211 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:17:33.139108 9211 webserver.cc:460] Webserver started at http://127.28.25.2:33203/ using document root <none> and password file <none>
I20260811 11:17:33.139389 9211 fs_manager.cc:359] Metadata directory not provided
I20260811 11:17:33.139456 9211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:17:33.139601 9211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:17:33 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:17:33.162068 9211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "99b3bbb8af2b4b26b7b991ab640c48df"
format_stamp: "Formatted at 2026-08-11 11:17:33 on dist-test-slave-2w3w"
server_key: "85b2d45a764f832162816306726ba375"
server_key_iv: "70b2448afff79e9b81c70f2f22911069"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:33.162581 9211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "99b3bbb8af2b4b26b7b991ab640c48df"
format_stamp: "Formatted at 2026-08-11 11:17:33 on dist-test-slave-2w3w"
server_key: "85b2d45a764f832162816306726ba375"
server_key_iv: "70b2448afff79e9b81c70f2f22911069"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:33.167227 9211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:17:33.187004 9232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:17:33.187364 9211 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260811 11:17:33.187522 9211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "99b3bbb8af2b4b26b7b991ab640c48df"
format_stamp: "Formatted at 2026-08-11 11:17:33 on dist-test-slave-2w3w"
server_key: "85b2d45a764f832162816306726ba375"
server_key_iv: "70b2448afff79e9b81c70f2f22911069"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:33.187646 9211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:17:33.205881 9211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:17:33.208667 9211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:17:33.208863 9211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:17:33.209169 9211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:17:33.209609 9211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:17:33.209667 9211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:17:33.209703 9211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:17:33.209722 9211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:17:33.217754 9211 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:36613
I20260811 11:17:33.218349 9211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:17:33.219304 9358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:36613 every 8 connection(s)
I20260811 11:17:33.220037 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 9211
I20260811 11:17:33.220152 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:17:33.220377 28772 external_mini_cluster.cc:1428] Setting key af98fe705c65a90b48ab492c5841895f
Aug 11 11:17:33 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:17:33.230983 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.219604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:58479 (local address 127.28.25.62:46647)
0811 11:17:33.219669 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:33.219671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:33.219690 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:17:33.220423 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:33.220424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:33.220437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:33.220474 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:17:33.221372 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.222187 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.223871 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.224210 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.224253 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.226938 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:17:33.226939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:17:33.226940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:17:33.226949 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:17:33.228916 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:17:33.229560 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:17:33.229561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:17:33.229562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:17:33.229634 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:17:33.230007 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:17:33.230008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:17:33.230008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:17:33.230171 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:17:33.230220 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:33.230691 (+ 471us) server_negotiation.cc:299] Negotiation successful
0811 11:17:33.230810 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 11:17:33.231573 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.219238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46647 (local address 127.28.25.2:58479)
0811 11:17:33.219526 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0811 11:17:33.219575 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 11:17:33.220290 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:17:33.220680 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:17:33.220684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:17:33.220973 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:17:33.221259 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.221265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.222318 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.222319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.223760 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.223764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.224349 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.224350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.224494 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.225127 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:17:33.225138 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:17:33.226791 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:17:33.229045 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:17:33.229058 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:17:33.229061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:17:33.229434 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:17:33.229718 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:17:33.229719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:17:33.229719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:17:33.229894 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:17:33.230277 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:17:33.230287 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:17:33.230506 (+ 219us) client_negotiation.cc:769] Sending connection context
0811 11:17:33.231103 (+ 597us) client_negotiation.cc:240] Negotiation successful
0811 11:17:33.231312 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":127,"threads_started":1}
I20260811 11:17:33.232102 9359 heartbeater.cc:347] Connected to a master server at 127.28.25.62:46647
I20260811 11:17:33.232220 9359 heartbeater.cc:464] Registering TS with master...
I20260811 11:17:33.232445 9359 heartbeater.cc:511] Master 127.28.25.62:46647 requested a full tablet report, sending...
I20260811 11:17:33.232832 8966 ts_manager.cc:194] Registered new tserver with Master: 99b3bbb8af2b4b26b7b991ab640c48df (127.28.25.2:36613)
I20260811 11:17:33.233345 8966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:58479
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:17:33.262465 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:54507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46647
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:36903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:17:33.400861 9365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:17:33.401193 9365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:17:33.401240 9365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:17:33.403252 9365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:17:33.403342 9365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:17:33.403391 9365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:17:33.406006 9365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.25.20:54507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:17:33.406361 9365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:17:33.406672 9365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:17:33.412600 9374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:17:33.412674 9371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:17:33.412714 9372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:17:33.413522 9365 server_base.cc:1034] running on GCE node
I20260811 11:17:33.413772 9365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:17:33.414160 9365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:17:33.415316 9365 hybrid_clock.cc:648] HybridClock initialized: now 1786447053415298 us; error 29 us; skew 500 ppm
Aug 11 11:17:33 dist-test-slave-2w3w krb5kdc[8437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:17:33.425293 9365 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:17:33.425820 9365 webserver.cc:460] Webserver started at http://127.28.25.3:42617/ using document root <none> and password file <none>
I20260811 11:17:33.426134 9365 fs_manager.cc:359] Metadata directory not provided
I20260811 11:17:33.426229 9365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:17:33.426385 9365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:17:33 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:17:33.444885 9365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "6d40d1eedece414b80614c1de772fa0e"
format_stamp: "Formatted at 2026-08-11 11:17:33 on dist-test-slave-2w3w"
server_key: "933d89f819567e995340d07b1642364c"
server_key_iv: "150850e92357ec9a2b3ce13837d80460"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:33.445343 9365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "6d40d1eedece414b80614c1de772fa0e"
format_stamp: "Formatted at 2026-08-11 11:17:33 on dist-test-slave-2w3w"
server_key: "933d89f819567e995340d07b1642364c"
server_key_iv: "150850e92357ec9a2b3ce13837d80460"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:33.448588 9365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:17:33.463488 9382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:17:33.463685 9365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:17:33.463792 9365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "6d40d1eedece414b80614c1de772fa0e"
format_stamp: "Formatted at 2026-08-11 11:17:33 on dist-test-slave-2w3w"
server_key: "933d89f819567e995340d07b1642364c"
server_key_iv: "150850e92357ec9a2b3ce13837d80460"
server_key_version: "kuduclusterkey@0"
I20260811 11:17:33.463909 9365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:17:33.483070 9365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:17:33.485900 9365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:17:33.486167 9365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:17:33.486408 9365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:17:33.486820 9365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:17:33.486876 9365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:17:33.486918 9365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:17:33.487018 9365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:17:33.495270 9365 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:36065
I20260811 11:17:33.495764 9365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:17:33.500149 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 9365
I20260811 11:17:33.500288 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:17:33.500492 28772 external_mini_cluster.cc:1428] Setting key b917a3d2337c54b3796afa513c681c66
Aug 11 11:17:33 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:17:33.507813 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.496627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:35539 (local address 127.28.25.62:46647)
0811 11:17:33.496679 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:33.496682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:33.496955 (+ 273us) server_negotiation.cc:407] Connection header received
0811 11:17:33.497893 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:33.497895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:33.497911 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:33.497953 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:17:33.499076 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.499897 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.502086 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.502316 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.502400 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.504634 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:17:33.504636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:17:33.504636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:17:33.504651 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:17:33.506123 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:17:33.506598 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:17:33.506599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:17:33.506600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:17:33.506663 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:17:33.507045 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:17:33.507047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:17:33.507047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:17:33.507203 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:17:33.507288 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:33.507619 (+ 331us) server_negotiation.cc:299] Negotiation successful
0811 11:17:33.507711 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:17:33.499569 9510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:36065 every 8 connection(s)
I20260811 11:17:33.508257 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.496633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46647 (local address 127.28.25.3:35539)
0811 11:17:33.496808 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 11:17:33.496855 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:17:33.497729 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:17:33.498190 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:17:33.498195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:17:33.498547 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:17:33.498899 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.498907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.500761 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.500762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.501979 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.501984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.502410 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.502411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.502562 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.503181 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:17:33.503190 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:17:33.504521 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:17:33.506256 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:17:33.506263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:17:33.506265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:17:33.506513 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:17:33.506738 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:17:33.506739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:17:33.506740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:17:33.506862 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:17:33.507293 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:17:33.507301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:17:33.507471 (+ 170us) client_negotiation.cc:769] Sending connection context
0811 11:17:33.507586 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 11:17:33.507755 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260811 11:17:33.512231 9511 heartbeater.cc:347] Connected to a master server at 127.28.25.62:46647
I20260811 11:17:33.512395 9511 heartbeater.cc:464] Registering TS with master...
I20260811 11:17:33.512622 9511 heartbeater.cc:511] Master 127.28.25.62:46647 requested a full tablet report, sending...
I20260811 11:17:33.513042 8965 ts_manager.cc:194] Registered new tserver with Master: 6d40d1eedece414b80614c1de772fa0e (127.28.25.3:36065)
I20260811 11:17:33.513450 8965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:35539
I20260811 11:17:33.523898 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:17:33.532627 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.525750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.28.25.62:46647)
0811 11:17:33.525798 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:33.525800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:33.525810 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:17:33.525886 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:33.525887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:33.525897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:33.525958 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:17:33.526598 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.527209 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.527993 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.528192 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.528265 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.529644 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:17:33.529646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:17:33.529646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:17:33.529654 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:17:33.531302 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:17:33.531724 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:17:33.531725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:17:33.531726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:17:33.531785 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:17:33.531989 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:17:33.531989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:17:33.531990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:17:33.532123 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:17:33.532198 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:33.532435 (+ 237us) server_negotiation.cc:299] Negotiation successful
0811 11:17:33.532521 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:17:33.535372 8965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:17:33.537724 8965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:17:33.548405 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.541827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:40533 (local address 127.0.0.1:40692)
0811 11:17:33.543038 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0811 11:17:33.543046 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 11:17:33.543122 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:17:33.543418 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:17:33.543421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:17:33.543423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:17:33.543500 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.543502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.545228 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.545229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.546950 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.546953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.548075 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.548076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.548162 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.548176 (+ 14us) client_negotiation.cc:769] Sending connection context
0811 11:17:33.548241 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:17:33.548309 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"thread_start_us":71,"threads_started":1}
I20260811 11:17:33.548977 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.542079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.28.25.1:40533)
0811 11:17:33.542792 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:33.542799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:33.543096 (+ 297us) server_negotiation.cc:407] Connection header received
0811 11:17:33.543202 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:33.543205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:33.543230 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:33.543300 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:17:33.543584 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.545116 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.547081 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.547891 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.547945 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.548006 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:33.548847 (+ 841us) server_negotiation.cc:299] Negotiation successful
0811 11:17:33.548899 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":56,"threads_started":1}
I20260811 11:17:33.550310 9132 tablet_service.cc:1467] Processing CreateTablet for tablet f51cebb381a3486c8b63a7d3b69d2d32 (DEFAULT_TABLE table=test-table [id=6c883edf2fc1441c818e520ace9e9591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:17:33.550745 9132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51cebb381a3486c8b63a7d3b69d2d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:17:33.554621 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.541131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:36065 (local address 127.0.0.1:38308)
0811 11:17:33.542541 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0811 11:17:33.542589 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:17:33.542656 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:17:33.550180 (+ 7524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:17:33.550182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:17:33.550185 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:17:33.550237 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.550238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.551764 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.551765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.553003 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.553006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.554382 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.554383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.554446 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.554451 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:17:33.554482 (+ 31us) client_negotiation.cc:240] Negotiation successful
0811 11:17:33.554529 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1377,"thread_start_us":197,"threads_started":1}
I20260811 11:17:33.555294 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.541241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.28.25.3:36065)
0811 11:17:33.549943 (+ 8702us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:33.549948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:33.549959 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:17:33.550000 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:33.550001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:33.550019 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:33.550329 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:17:33.550419 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.551651 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.553469 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.554256 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.555038 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.555131 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:33.555159 (+ 28us) server_negotiation.cc:299] Negotiation successful
0811 11:17:33.555198 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8655,"thread_start_us":118,"threads_started":1}
I20260811 11:17:33.556123 9436 tablet_service.cc:1467] Processing CreateTablet for tablet f51cebb381a3486c8b63a7d3b69d2d32 (DEFAULT_TABLE table=test-table [id=6c883edf2fc1441c818e520ace9e9591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:17:33.556489 9436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51cebb381a3486c8b63a7d3b69d2d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:17:33.563177 9529 tablet_bootstrap.cc:492] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707: Bootstrap starting.
I20260811 11:17:33.564111 9529 tablet_bootstrap.cc:654] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707: Neither blocks nor log segments found. Creating new log.
I20260811 11:17:33.564476 9529 log.cc:824] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707: Log is configured to *not* fsync() on all Append() calls
I20260811 11:17:33.571177 9531 tablet_bootstrap.cc:492] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: Bootstrap starting.
I20260811 11:17:33.571409 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.541688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.28.25.2:36613)
0811 11:17:33.567098 (+ 25410us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:33.567104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:33.567120 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:17:33.567167 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:33.567170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:33.567188 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:33.567498 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:17:33.567583 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.568677 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.570145 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.570945 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.571066 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.571164 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:33.571278 (+ 114us) server_negotiation.cc:299] Negotiation successful
0811 11:17:33.571330 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25358,"thread_start_us":103,"threads_started":1}
I20260811 11:17:33.571439 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.541545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:36613 (local address 127.0.0.1:36012)
0811 11:17:33.542849 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0811 11:17:33.542858 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:17:33.542899 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:17:33.567347 (+ 24448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:17:33.567348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:17:33.567353 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:17:33.567423 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.567426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.568800 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.568802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.569938 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.569943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.571081 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.571083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.571170 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.571177 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:17:33.571260 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 11:17:33.571327 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"thread_start_us":138,"threads_started":1}
I20260811 11:17:33.572425 9531 tablet_bootstrap.cc:654] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: Neither blocks nor log segments found. Creating new log.
I20260811 11:17:33.572283 9266 tablet_service.cc:1467] Processing CreateTablet for tablet f51cebb381a3486c8b63a7d3b69d2d32 (DEFAULT_TABLE table=test-table [id=6c883edf2fc1441c818e520ace9e9591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:17:33.572690 9266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51cebb381a3486c8b63a7d3b69d2d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:17:33.572796 9531 log.cc:824] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: Log is configured to *not* fsync() on all Append() calls
I20260811 11:17:33.573547 9529 tablet_bootstrap.cc:492] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707: No bootstrap required, opened a new log
I20260811 11:17:33.573640 9529 ts_tablet_manager.cc:1397] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260811 11:17:33.575433 9531 tablet_bootstrap.cc:492] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: No bootstrap required, opened a new log
I20260811 11:17:33.575536 9531 ts_tablet_manager.cc:1397] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 11:17:33.576715 9529 raft_consensus.cc:348] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 } } peers { permanent_uuid: "12c661f457fc49d38289428d1feff707" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 40533 } } peers { permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 } }
I20260811 11:17:33.576934 9529 raft_consensus.cc:374] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:17:33.576973 9529 raft_consensus.cc:729] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c661f457fc49d38289428d1feff707, State: Initialized, Role: FOLLOWER
I20260811 11:17:33.577128 9529 consensus_queue.cc:260] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 } } peers { permanent_uuid: "12c661f457fc49d38289428d1feff707" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 40533 } } peers { permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 } }
I20260811 11:17:33.577423 9529 ts_tablet_manager.cc:1428] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 11:17:33.577554 9205 heartbeater.cc:503] Master 127.28.25.62:46647 was elected leader, sending a full tablet report...
I20260811 11:17:33.577533 9531 raft_consensus.cc:348] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 } } peers { permanent_uuid: "12c661f457fc49d38289428d1feff707" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 40533 } } peers { permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 } }
I20260811 11:17:33.577744 9531 raft_consensus.cc:374] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:17:33.577800 9531 raft_consensus.cc:729] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d40d1eedece414b80614c1de772fa0e, State: Initialized, Role: FOLLOWER
I20260811 11:17:33.577991 9531 consensus_queue.cc:260] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 } } peers { permanent_uuid: "12c661f457fc49d38289428d1feff707" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 40533 } } peers { permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 } }
I20260811 11:17:33.578289 9531 ts_tablet_manager.cc:1428] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:17:33.578423 9511 heartbeater.cc:503] Master 127.28.25.62:46647 was elected leader, sending a full tablet report...
I20260811 11:17:33.580161 9534 tablet_bootstrap.cc:492] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: Bootstrap starting.
I20260811 11:17:33.581415 9534 tablet_bootstrap.cc:654] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: Neither blocks nor log segments found. Creating new log.
I20260811 11:17:33.581816 9534 log.cc:824] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: Log is configured to *not* fsync() on all Append() calls
I20260811 11:17:33.582908 9534 tablet_bootstrap.cc:492] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: No bootstrap required, opened a new log
I20260811 11:17:33.583035 9534 ts_tablet_manager.cc:1397] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:17:33.585327 9534 raft_consensus.cc:348] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 } } peers { permanent_uuid: "12c661f457fc49d38289428d1feff707" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 40533 } } peers { permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 } }
I20260811 11:17:33.585552 9534 raft_consensus.cc:374] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:17:33.585603 9534 raft_consensus.cc:729] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b3bbb8af2b4b26b7b991ab640c48df, State: Initialized, Role: FOLLOWER
I20260811 11:17:33.585855 9534 consensus_queue.cc:260] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 } } peers { permanent_uuid: "12c661f457fc49d38289428d1feff707" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 40533 } } peers { permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 } }
I20260811 11:17:33.586331 9534 ts_tablet_manager.cc:1428] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 11:17:33.586474 9359 heartbeater.cc:503] Master 127.28.25.62:46647 was elected leader, sending a full tablet report...
I20260811 11:17:33.670182 9535 raft_consensus.cc:482] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:17:33.670372 9535 raft_consensus.cc:504] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 } } peers { permanent_uuid: "12c661f457fc49d38289428d1feff707" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 40533 } } peers { permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 } }
I20260811 11:17:33.670817 9535 leader_election.cc:288] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99b3bbb8af2b4b26b7b991ab640c48df (127.28.25.2:36613), 6d40d1eedece414b80614c1de772fa0e (127.28.25.3:36065)
W20260811 11:17:33.673821 9206 tablet.cc:2369] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:17:33.678606 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.671220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:46119 (local address 127.28.25.3:36065)
0811 11:17:33.671270 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:33.671273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:33.671788 (+ 515us) server_negotiation.cc:407] Connection header received
0811 11:17:33.671900 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:33.671902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:33.671912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:33.671944 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:17:33.672203 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.672927 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.675910 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.676478 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.676530 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.676543 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:33.678470 (+ 1927us) server_negotiation.cc:299] Negotiation successful
0811 11:17:33.678524 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 11:17:33.678913 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.671148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:36065 (local address 127.28.25.1:46119)
0811 11:17:33.671714 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0811 11:17:33.671725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:17:33.671818 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:17:33.672018 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:17:33.672019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:17:33.672021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:17:33.672128 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.672130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.673064 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.673065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.674410 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.674414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.678284 (+ 3870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.678285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.678372 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.678376 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:17:33.678773 (+ 397us) client_negotiation.cc:240] Negotiation successful
0811 11:17:33.678827 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":130,"threads_started":1}
I20260811 11:17:33.679493 9458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51cebb381a3486c8b63a7d3b69d2d32" candidate_uuid: "12c661f457fc49d38289428d1feff707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d40d1eedece414b80614c1de772fa0e" is_pre_election: true
I20260811 11:17:33.679675 9458 raft_consensus.cc:2454] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c661f457fc49d38289428d1feff707 in term 0.
I20260811 11:17:33.679961 9080 leader_election.cc:302] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c661f457fc49d38289428d1feff707, 6d40d1eedece414b80614c1de772fa0e; no voters:
I20260811 11:17:33.680099 9535 raft_consensus.cc:2781] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:17:33.680143 9535 raft_consensus.cc:482] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:17:33.680166 9535 raft_consensus.cc:3037] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:17:33.681152 9535 raft_consensus.cc:504] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 } } peers { permanent_uuid: "12c661f457fc49d38289428d1feff707" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 40533 } } peers { permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 } }
I20260811 11:17:33.681301 9535 leader_election.cc:288] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [CANDIDATE]: Term 1 election: Requested vote from peers 99b3bbb8af2b4b26b7b991ab640c48df (127.28.25.2:36613), 6d40d1eedece414b80614c1de772fa0e (127.28.25.3:36065)
I20260811 11:17:33.681916 9458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51cebb381a3486c8b63a7d3b69d2d32" candidate_uuid: "12c661f457fc49d38289428d1feff707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d40d1eedece414b80614c1de772fa0e"
I20260811 11:17:33.682008 9458 raft_consensus.cc:3037] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:17:33.683068 9458 raft_consensus.cc:2454] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c661f457fc49d38289428d1feff707 in term 1.
I20260811 11:17:33.683291 9080 leader_election.cc:302] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c661f457fc49d38289428d1feff707, 6d40d1eedece414b80614c1de772fa0e; no voters:
I20260811 11:17:33.683441 9535 raft_consensus.cc:2781] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:17:33.683624 9535 raft_consensus.cc:686] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [term 1 LEADER]: Becoming Leader. State: Replica: 12c661f457fc49d38289428d1feff707, State: Running, Role: LEADER
I20260811 11:17:33.683732 9535 consensus_queue.cc:237] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 } } peers { permanent_uuid: "12c661f457fc49d38289428d1feff707" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 40533 } } peers { permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 } }
I20260811 11:17:33.684769 8965 catalog_manager.cc:5495] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12c661f457fc49d38289428d1feff707 (127.28.25.1). New cstate: current_term: 1 leader_uuid: "12c661f457fc49d38289428d1feff707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c661f457fc49d38289428d1feff707" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 40533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 } health_report { overall_health: UNKNOWN } } }
I20260811 11:17:33.687565 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.671607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:50541 (local address 127.28.25.2:36613)
0811 11:17:33.671647 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:33.671649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:33.674607 (+ 2958us) server_negotiation.cc:407] Connection header received
0811 11:17:33.674731 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:33.674732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:33.674747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:33.674778 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:17:33.675032 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.675763 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.686449 (+ 10686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.687134 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.687192 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.687208 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:33.687446 (+ 238us) server_negotiation.cc:299] Negotiation successful
0811 11:17:33.687489 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 11:17:33.688129 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.671397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:36613 (local address 127.28.25.1:50541)
0811 11:17:33.674546 (+ 3149us) negotiation.cc:107] Waiting for socket to connect
0811 11:17:33.674558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:17:33.674638 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:17:33.674849 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:17:33.674850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:17:33.674852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:17:33.674924 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.674925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.676609 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.676610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.686319 (+ 9709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:17:33.686323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.687279 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.687280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:17:33.687361 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.687365 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:17:33.687963 (+ 598us) client_negotiation.cc:240] Negotiation successful
0811 11:17:33.688020 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3128,"thread_start_us":92,"threads_started":1}
I20260811 11:17:33.688692 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51cebb381a3486c8b63a7d3b69d2d32" candidate_uuid: "12c661f457fc49d38289428d1feff707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" is_pre_election: true
I20260811 11:17:33.688875 9309 raft_consensus.cc:2454] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c661f457fc49d38289428d1feff707 in term 0.
I20260811 11:17:33.688997 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51cebb381a3486c8b63a7d3b69d2d32" candidate_uuid: "12c661f457fc49d38289428d1feff707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99b3bbb8af2b4b26b7b991ab640c48df"
I20260811 11:17:33.689052 9308 raft_consensus.cc:3037] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:17:33.690043 9308 raft_consensus.cc:2454] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c661f457fc49d38289428d1feff707 in term 1.
I20260811 11:17:33.696882 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.693159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.28.25.1:40533)
0811 11:17:33.693207 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 11:17:33.693209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:17:33.693221 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:17:33.693313 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:17:33.693314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:17:33.693341 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:17:33.693427 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:17:33.693659 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.694400 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.695571 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:17:33.695863 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:17:33.695941 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:17:33.696165 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:17:33.696341 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:17:33.696449 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 11:17:33.696733 (+ 284us) server_negotiation.cc:299] Negotiation successful
0811 11:17:33.696767 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:17:33.700253 9458 raft_consensus.cc:1261] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e [term 1 FOLLOWER]: Refusing update from remote peer 12c661f457fc49d38289428d1feff707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:17:33.700775 9541 consensus_queue.cc:1035] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d40d1eedece414b80614c1de772fa0e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 36065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:17:33.704039 9308 raft_consensus.cc:1261] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df [term 1 FOLLOWER]: Refusing update from remote peer 12c661f457fc49d38289428d1feff707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:17:33.704530 9541 consensus_queue.cc:1035] T f51cebb381a3486c8b63a7d3b69d2d32 P 12c661f457fc49d38289428d1feff707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99b3bbb8af2b4b26b7b991ab640c48df" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:17:33.717940 9544 mvcc.cc:205] Tried to move back new op lower bound from 7317287131952287744 to 7317287131888889856. Current Snapshot: MvccSnapshot[applied={T|T < 7317287131952287744}]
I20260811 11:17:33.718211 9545 mvcc.cc:205] Tried to move back new op lower bound from 7317287131952287744 to 7317287131888889856. Current Snapshot: MvccSnapshot[applied={T|T < 7317287131952287744}]
W20260811 11:17:33.720175 9360 tablet.cc:2369] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:17:33.721899 9547 mvcc.cc:205] Tried to move back new op lower bound from 7317287131952287744 to 7317287131888889856. Current Snapshot: MvccSnapshot[applied={T|T < 7317287131952287744}]
I20260811 11:17:33.737156 8965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:17:33.737303 8965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:17:33.738291 8965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca32c346dd5442d5ad72064ec0d5594c: Sending DeleteTablet for 3 replicas of tablet f51cebb381a3486c8b63a7d3b69d2d32
I20260811 11:17:33.738744 9436 tablet_service.cc:1514] Processing DeleteTablet for tablet f51cebb381a3486c8b63a7d3b69d2d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:17:33 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:38308
I20260811 11:17:33.739434 9266 tablet_service.cc:1514] Processing DeleteTablet for tablet f51cebb381a3486c8b63a7d3b69d2d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:17:33 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:36012
I20260811 11:17:33.739671 9132 tablet_service.cc:1514] Processing DeleteTablet for tablet f51cebb381a3486c8b63a7d3b69d2d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:17:33 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:40692
I20260811 11:17:33.740456 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 9058
Aug 11 11:17:33 dist-test-slave-2w3w krb5kdc[8437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:17:33.758852 9555 tablet_replica.cc:330] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: stopping tablet replica
I20260811 11:17:33.767169 9555 raft_consensus.cc:2229] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:17:33.767560 9555 raft_consensus.cc:2258] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:17:33.767918 9553 tablet_replica.cc:330] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: stopping tablet replica
I20260811 11:17:33.768496 9555 ts_tablet_manager.cc:1905] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:17:33.768720 9553 raft_consensus.cc:2229] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:17:33.768950 9553 raft_consensus.cc:2258] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:17:33.772261 9555 ts_tablet_manager.cc:1918] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:17:33.772455 9555 log.cc:1196] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/f51cebb381a3486c8b63a7d3b69d2d32
I20260811 11:17:33.772603 9553 ts_tablet_manager.cc:1905] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:17:33.772867 9555 ts_tablet_manager.cc:1939] T f51cebb381a3486c8b63a7d3b69d2d32 P 99b3bbb8af2b4b26b7b991ab640c48df: Deleting consensus metadata
I20260811 11:17:33.773396 8950 catalog_manager.cc:4841] TS 99b3bbb8af2b4b26b7b991ab640c48df (127.28.25.2:36613): tablet f51cebb381a3486c8b63a7d3b69d2d32 (table test-table [id=6c883edf2fc1441c818e520ace9e9591]) successfully deleted
W20260811 11:17:33.775774 8950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:40533 (error 108)
I20260811 11:17:33.776250 9553 ts_tablet_manager.cc:1918] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:17:33.776458 9553 log.cc:1196] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/f51cebb381a3486c8b63a7d3b69d2d32
I20260811 11:17:33.777349 9553 ts_tablet_manager.cc:1939] T f51cebb381a3486c8b63a7d3b69d2d32 P 6d40d1eedece414b80614c1de772fa0e: Deleting consensus metadata
I20260811 11:17:33.777927 8949 catalog_manager.cc:4841] TS 6d40d1eedece414b80614c1de772fa0e (127.28.25.3:36065): tablet f51cebb381a3486c8b63a7d3b69d2d32 (table test-table [id=6c883edf2fc1441c818e520ace9e9591]) successfully deleted
I20260811 11:17:33.786935 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 9211
I20260811 11:17:33.792356 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:17:33.792170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:40533 (local address 127.0.0.1:40710)
0811 11:17:33.792249 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0811 11:17:33.792302 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260811 11:17:33.794699 8950 catalog_manager.cc:4584] TS 12c661f457fc49d38289428d1feff707 (127.28.25.1:40533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f51cebb381a3486c8b63a7d3b69d2d32: Network error: Client connection negotiation failed: client connection to 127.28.25.1:40533: connect: Connection refused (error 111)
I20260811 11:17:33.808151 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 9365
I20260811 11:17:33.823544 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 8929
I20260811 11:17:33.843526 28772 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 11:17:34.199323 28772 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 11:17:34.199445 28772 mini_ranger.cc:66] Stopping Ranger...
I20260811 11:17:34.555197 28772 mini_ranger.cc:68] Stopped Ranger
2026-08-11 11:17:34.555 UTC [8471] LOG: received smart shutdown request
2026-08-11 11:17:34.560 UTC [8471] LOG: background worker "logical replication launcher" (PID 8482) exited with exit code 1
2026-08-11 11:17:34.560 UTC [8475] LOG: shutting down
2026-08-11 11:17:34.764 UTC [8471] LOG: database system is shut down
2026-08-11T11:17:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54712 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:17:34 dist-test-slave-2w3w krb5kdc[9568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:17:34 dist-test-slave-2w3w krb5kdc[9568](info): set up 2 sockets
Aug 11 11:17:34 dist-test-slave-2w3w krb5kdc[9568](info): commencing operation
krb5kdc: starting...
W20260811 11:17:36.041451 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:17:36 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:17:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:17:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.25.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.25.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/rangeradmin_127.28.25.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/rangeradmin_127.28.25.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.25.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.25.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/rangerlookup_127.28.25.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/rangerlookup_127.28.25.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.25.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.25.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/HTTP_127.28.25.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/HTTP_127.28.25.20@KRBTEST.COM.keytab.
I20260811 11:17:36.263406 28772 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskR10UZg/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/postgres -l logfile start
2026-08-11 11:17:39.298 UTC [9602] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 11:17:39.298 UTC [9602] LOG: listening on IPv4 address "127.28.25.20", port 53295
2026-08-11 11:17:39.302 UTC [9602] LOG: listening on Unix socket "/tmp/.s.PGSQL.53295"
2026-08-11 11:17:39.325 UTC [9606] LOG: database system was shut down at 2026-08-11 11:17:37 UTC
2026-08-11 11:17:39.329 UTC [9602] LOG: database system is ready to accept connections
I20260811 11:17:40.383080 28772 mini_postgres.cc:96] Postgres bound to 53295
2026-08-11 11:17:40.391 UTC [9615] FATAL: database "slave" does not exist
127.28.25.20:53295 - accepting connections
I20260811 11:17:40.391858 28772 mini_ranger.cc:161] Starting Ranger...
I20260811 11:17:40.410637 28772 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 11:17:41.106073 28772 mini_ranger.cc:87] Created ranger Postgres database
I20260811 11:17:41.106187 28772 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-admin
2026-08-11 11:17:41,415 [I] DB FLAVOR :POSTGRES
2026-08-11 11:17:41,415 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 11:17:41,415 [I] Checking connection..
2026-08-11 11:17:41,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 11:17:41,741 [I] Checking connection passed.
2026-08-11 11:17:41,742 [I] --------- Verifying version history table ---------
2026-08-11 11:17:41,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:17:42,031 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 11:17:42,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:17:42,315 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 11:17:42,315 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 11:17:42,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 11:17:42.602 UTC [9688] WARNING: there is no transaction in progress
2026-08-11 11:17:42,611 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 11:17:42,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:17:42,898 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 11:17:42,899 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 11:17:42,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 11:17:43,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 11:17:43,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2w3w.c.gcp-upstream.internal','N') ;"
2026-08-11 11:17:43,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 11:17:44,043 [I] Table x_portal_user does not exist in database ranger
2026-08-11 11:17:44,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 11:17:44,328 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 11:17:44,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 11:17:44,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 11:17:44,894 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 11:17:44,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 11:17:46.027 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.046 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.066 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.085 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.104 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.126 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.190 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.199 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.215 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.226 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.238 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.254 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.265 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.274 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.283 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:46.292 UTC [9812] WARNING: there is no transaction in progress
2026-08-11 11:17:47,183 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 11:17:47,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 11:17:47,458 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 11:17:47,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 11:17:47,787 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 11:17:47,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 11:17:48,133 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 11:17:48,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 11:17:48,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 11:17:48,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2w3w.c.gcp-upstream.internal';"
2026-08-11 11:17:49,157 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 11:17:49,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 11:17:49,457 [I] DB_PATCHES have already been applied
I20260811 11:17:49.463838 28772 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-admin:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskR10UZg/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 11:17:49.463976 28772 mini_ranger.cc:193] Using host: 127.28.25.20
I20260811 11:17:49.468601 28772 mini_ranger.cc:227] Ranger admin URL: http://127.28.25.20:38287
Aug 11, 2026 11:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 11:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 11:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 11:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 11:17:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 11:17:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 11:17:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 11:17:50.673391 28772 mini_ranger.cc:160] Time spent starting Ranger: real 10.282s user 0.001s sys 0.012s
Aug 11, 2026 11:18:14 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 11:18:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 11:18:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 11:18:14 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 11:18:15 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 11:18:16.415179 28772 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.25.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.25.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/rangerkms_127.28.25.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.25.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/rangerkms_127.28.25.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.25.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.25.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.25.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/HTTP_127.28.25.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.25.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/HTTP_127.28.25.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 11:18:16 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447096, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 11:18:16.521486 28772 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 11:18:16.535668 28772 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 11:18:17.925661 28772 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 11:18:17.925762 28772 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-kms
I20260811 11:18:17.925781 28772 mini_ranger_kms.cc:227] Using postgres at 127.28.25.20:53295
2026-08-11 11:18:18,123 [I] DB FLAVOR :POSTGRES
2026-08-11 11:18:18,123 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 11:18:18,124 [I] Checking connection
2026-08-11 11:18:18,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 11:18:18,410 [I] connection success
2026-08-11 11:18:18,410 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 11:18:18,411 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 11:18:18,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 11:18:18,678 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 11:18:18,679 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 11:18:18,679 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 11:18:18,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.25.20:53295/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 11:18:19,020 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 11:18:19.635592 28772 mini_ranger_kms.cc:310] Created kms service
I20260811 11:18:19.874221 28772 mini_ranger_kms.cc:326] Created kudu user
I20260811 11:18:19.929692 28772 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 11:18:19 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447096, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:18:20.429352 28772 mini_ranger_kms.cc:384] Added ranger policy
I20260811 11:18:20.429482 28772 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-kms:/tmp/dist-test-taskR10UZg/build/release/bin/postgresql.jar:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskR10UZg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 11:18:20.429503 28772 mini_ranger_kms.cc:240] Using host: 127.28.25.20
I20260811 11:18:20.434659 28772 mini_ranger_kms.cc:276] Ranger KMS PID: 10023
I20260811 11:18:20.434805 28772 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.25.20:48961
Aug 11, 2026 11:18:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 11:18:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 11:18:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 11:18:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 11:18:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 11:18:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260811 11:18:22.994504 28772 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.473s user 0.000s sys 0.181s
I20260811 11:18:22.994620 28772 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 11:18:22.994673 28772 mini_ranger_kms.cc:401] 127.28.25.20:48961/kms/v1/keys
Aug 11 11:18:23 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786447103, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.25.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 11 11:18:23 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447103, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.25.20@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
Aug 11 11:18:24 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447103, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.25.20@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
Aug 11 11:18:26 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:18:26.803136 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:44173
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:44173
--ranger_config_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:48961/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:26.903743 10080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:26.904047 10080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:26.904096 10080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:26.905689 10080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:18:26.905786 10080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:26.905823 10080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:26.905861 10080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:18:26.905889 10080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:18:26.908018 10080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.25.20:48961/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:44173
--ranger_config_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:26.908392 10080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:26.908682 10080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:18:26.916244 10080 server_base.cc:1034] running on GCE node
W20260811 11:18:26.916272 10085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:26.916548 10088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:26.916375 10086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:26.916970 10080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:26.917376 10080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:26.918598 10080 hybrid_clock.cc:648] HybridClock initialized: now 1786447106918573 us; error 48 us; skew 500 ppm
Aug 11 11:18:26 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:26.926280 10080 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:18:26.926916 10080 webserver.cc:460] Webserver started at http://127.28.25.62:45007/ using document root <none> and password file <none>
I20260811 11:18:26.927260 10080 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:26.927340 10080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:26.927523 10080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:18:26 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:18:26.981297 10080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "f532cb2964a64bca9f0a3dade5e06a6a"
format_stamp: "Formatted at 2026-08-11 11:18:26 on dist-test-slave-2w3w"
server_key: "ef8cd2aaec28cc91badeca51f165907b"
server_key_iv: "c0ca8ab36f9da1f5e17560f906091fd7"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:26.981763 10080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "f532cb2964a64bca9f0a3dade5e06a6a"
format_stamp: "Formatted at 2026-08-11 11:18:26 on dist-test-slave-2w3w"
server_key: "ef8cd2aaec28cc91badeca51f165907b"
server_key_iv: "c0ca8ab36f9da1f5e17560f906091fd7"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:26.983480 10080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:18:26.996657 10096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:26.996918 10080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:26.997030 10080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "f532cb2964a64bca9f0a3dade5e06a6a"
format_stamp: "Formatted at 2026-08-11 11:18:26 on dist-test-slave-2w3w"
server_key: "ef8cd2aaec28cc91badeca51f165907b"
server_key_iv: "c0ca8ab36f9da1f5e17560f906091fd7"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:26.997136 10080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:27.009461 10080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:27.012327 10080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:27.012720 10080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:27.018395 10080 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:44173
I20260811 11:18:27.018424 10154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:44173 every 8 connection(s)
I20260811 11:18:27.018893 10080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:18:27.019799 10155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:27.020710 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 10080
I20260811 11:18:27.020833 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:18:27.021052 28772 external_mini_cluster.cc:1428] Setting key c5a6f880c602e6bb90f4e07bdb4fba51
I20260811 11:18:27.023434 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: Bootstrap starting.
I20260811 11:18:27.024330 10155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:27.024673 10155 log.cc:824] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:27.025591 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: No bootstrap required, opened a new log
Aug 11 11:18:27 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:27.028481 10155 raft_consensus.cc:348] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } }
I20260811 11:18:27.028728 10155 raft_consensus.cc:374] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:27.028821 10155 raft_consensus.cc:729] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f532cb2964a64bca9f0a3dade5e06a6a, State: Initialized, Role: FOLLOWER
I20260811 11:18:27.028985 10155 consensus_queue.cc:260] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } }
I20260811 11:18:27.029084 10155 raft_consensus.cc:388] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:18:27.029119 10155 raft_consensus.cc:482] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:18:27.029162 10155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:27.029942 10155 raft_consensus.cc:504] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } }
I20260811 11:18:27.030084 10155 leader_election.cc:302] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f532cb2964a64bca9f0a3dade5e06a6a; no voters:
I20260811 11:18:27.030296 10155 leader_election.cc:288] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:18:27.030442 10161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:18:27.030614 10155 sys_catalog.cc:564] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:18:27.030651 10161 raft_consensus.cc:686] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 1 LEADER]: Becoming Leader. State: Replica: f532cb2964a64bca9f0a3dade5e06a6a, State: Running, Role: LEADER
I20260811 11:18:27.030848 10161 consensus_queue.cc:237] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } }
I20260811 11:18:27.031354 10161 sys_catalog.cc:455] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } } }
I20260811 11:18:27.031533 10161 sys_catalog.cc:458] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:27.031723 10161 sys_catalog.cc:455] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f532cb2964a64bca9f0a3dade5e06a6a. Latest consensus state: current_term: 1 leader_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } } }
I20260811 11:18:27.031836 10161 sys_catalog.cc:458] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:27.032032 10171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:18:27.032377 10155 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 11:18:27.032723 10171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:18:27.033000 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:27.021662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.28.25.62:44173)
0811 11:18:27.021855 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:27.021865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:27.021901 (+ 36us) server_negotiation.cc:407] Connection header received
0811 11:18:27.022407 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:27.022415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:27.022465 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:27.022623 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:27.023428 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:27.024514 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:27.025522 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:27.026022 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:27.026154 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:27.028660 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:27.028664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:27.028665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:27.028676 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:27.030730 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:27.031227 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:27.031231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:27.031232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:27.031339 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:27.031558 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:27.031559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:27.031559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:27.031916 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:27.032095 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:27.032640 (+ 545us) server_negotiation.cc:299] Negotiation successful
0811 11:18:27.032789 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260811 11:18:27.040082 10171 catalog_manager.cc:1347] Generated new cluster ID: 2a9fa90f05514341aa95135a75895cd0
I20260811 11:18:27.040190 10171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:18:27.076504 10171 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:18:27.077329 10171 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:18:27.082923 10171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: Generated new TSK 0
I20260811 11:18:27.083158 10171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:18:27.866714 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:48961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44173
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:27.997128 10206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:27.997459 10206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:27.997501 10206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:27.999486 10206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:27.999573 10206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:27.999621 10206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:18:28.002187 10206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.25.20:48961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:28.002547 10206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:28.002924 10206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:28.011300 10211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:28.011307 10212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:28.011544 10206 server_base.cc:1034] running on GCE node
W20260811 11:18:28.011307 10216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:28.012027 10206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:28.012363 10206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:28.013585 10206 hybrid_clock.cc:648] HybridClock initialized: now 1786447108013551 us; error 56 us; skew 500 ppm
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:28.017630 10206 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:18:28.018415 10206 webserver.cc:460] Webserver started at http://127.28.25.1:37729/ using document root <none> and password file <none>
I20260811 11:18:28.018708 10206 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:28.018790 10206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:28.019054 10206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:18:28.035710 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "4cdb667263c44e6ca18904092b4bde7d"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "a4efb0c133fa17f8209269388f8579de"
server_key_iv: "7d579b7417e6fee806b889e1551d7a4e"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:28.036249 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "4cdb667263c44e6ca18904092b4bde7d"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "a4efb0c133fa17f8209269388f8579de"
server_key_iv: "7d579b7417e6fee806b889e1551d7a4e"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:28.038331 10206 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:18:28.055164 10224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:28.055545 10206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:18:28.055665 10206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "4cdb667263c44e6ca18904092b4bde7d"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "a4efb0c133fa17f8209269388f8579de"
server_key_iv: "7d579b7417e6fee806b889e1551d7a4e"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:28.055774 10206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:28.072209 10206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:28.074880 10206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:28.075086 10206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:28.075373 10206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:28.075757 10206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:28.075802 10206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:28.075839 10206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:28.075863 10206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:28.083568 10206 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:38701
I20260811 11:18:28.083936 10351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:38701 every 8 connection(s)
I20260811 11:18:28.084059 10206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:18:28.086056 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 10206
I20260811 11:18:28.086155 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:18:28.086380 28772 external_mini_cluster.cc:1428] Setting key 8ec59aeb19d03dd20ab84312a5af53f4
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
I20260811 11:18:28.097892 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:28.085240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:44173 (local address 127.28.25.1:44097)
0811 11:18:28.085507 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:28.085539 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:28.086285 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:28.087154 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:28.087160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:28.087454 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:28.087757 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:28.087764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.088679 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.088680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:28.089975 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:28.089979 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.090532 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.090533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:28.090686 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:28.091403 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:28.091414 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:28.093379 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:28.095854 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:28.095875 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:28.095878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:28.096235 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:28.096492 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:28.096493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:28.096493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:28.096645 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:28.097020 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:28.097028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:28.097253 (+ 225us) client_negotiation.cc:769] Sending connection context
0811 11:18:28.097381 (+ 128us) client_negotiation.cc:240] Negotiation successful
0811 11:18:28.097566 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260811 11:18:28.098433 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:28.085260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:44097 (local address 127.28.25.62:44173)
0811 11:18:28.085389 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:28.085392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:28.085632 (+ 240us) server_negotiation.cc:407] Connection header received
0811 11:18:28.086474 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:28.086475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:28.086494 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:28.087009 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:28.087892 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.088555 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.090095 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.090416 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.090489 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:28.093524 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:28.093526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:28.093527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:28.093537 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:28.095735 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:28.096348 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:28.096350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:28.096351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:28.096416 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:28.096726 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:28.096728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:28.096728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:28.096913 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:28.096980 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:28.098211 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0811 11:18:28.098326 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260811 11:18:28.099140 10353 heartbeater.cc:347] Connected to a master server at 127.28.25.62:44173
I20260811 11:18:28.099275 10353 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:28.099509 10353 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
I20260811 11:18:28.100087 10116 ts_manager.cc:194] Registered new tserver with Master: 4cdb667263c44e6ca18904092b4bde7d (127.28.25.1:38701)
I20260811 11:18:28.100796 10116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:44097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:18:28.122702 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:48961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44173
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:28.241362 10360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:28.241689 10360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:28.241737 10360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:28.243655 10360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:28.243729 10360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:28.243775 10360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:18:28.246273 10360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.25.20:48961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:28.246578 10360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:28.246933 10360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:28.253875 10372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:28.253880 10370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:28.254336 10374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:28.254614 10360 server_base.cc:1034] running on GCE node
I20260811 11:18:28.254876 10360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:28.255209 10360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:28.256402 10360 hybrid_clock.cc:648] HybridClock initialized: now 1786447108256380 us; error 37 us; skew 500 ppm
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:28.259967 10360 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:18:28.260594 10360 webserver.cc:460] Webserver started at http://127.28.25.2:39787/ using document root <none> and password file <none>
I20260811 11:18:28.260955 10360 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:28.261034 10360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:28.261222 10360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:28.287946 10360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "9fb6928b7c7b475ba75ac78814ef7d9c"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "6981928be1c38f624cd5f8424f141ec3"
server_key_iv: "49e766002477d2c50eebad259144415e"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:28.288457 10360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "9fb6928b7c7b475ba75ac78814ef7d9c"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "6981928be1c38f624cd5f8424f141ec3"
server_key_iv: "49e766002477d2c50eebad259144415e"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:28.295490 10360 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.000s
I20260811 11:18:28.315961 10382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:28.316238 10360 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260811 11:18:28.316352 10360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "9fb6928b7c7b475ba75ac78814ef7d9c"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "6981928be1c38f624cd5f8424f141ec3"
server_key_iv: "49e766002477d2c50eebad259144415e"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:28.316483 10360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:28.331882 10360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:28.334472 10360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:28.334622 10360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:28.334879 10360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:28.335283 10360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:28.335328 10360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:28.335358 10360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:28.335393 10360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:28.342912 10360 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:43791
I20260811 11:18:28.343114 10515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:43791 every 8 connection(s)
I20260811 11:18:28.343410 10360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:18:28.350714 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 10360
I20260811 11:18:28.350832 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:18:28.351058 28772 external_mini_cluster.cc:1428] Setting key 43abb8a1cbe9a54866ffd268653e34e9
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:28.356678 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:28.344349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:43479 (local address 127.28.25.62:44173)
0811 11:18:28.344412 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:28.344415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:28.344667 (+ 252us) server_negotiation.cc:407] Connection header received
0811 11:18:28.345405 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:28.345407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:28.345421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:28.345453 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:28.347186 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.347978 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.349508 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.349820 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.349863 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:28.352914 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:28.352916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:28.352916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:28.352925 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:28.354734 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:28.355361 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:28.355362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:28.355363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:28.355441 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:28.355786 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:28.355787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:28.355787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:28.355945 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:28.356017 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:28.356464 (+ 447us) server_negotiation.cc:299] Negotiation successful
0811 11:18:28.356571 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:18:28.357504 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:28.344322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:44173 (local address 127.28.25.2:43479)
0811 11:18:28.344517 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:28.344570 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:28.345261 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:28.345647 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:28.345651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:28.345931 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:28.346236 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:28.346243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.348113 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.348114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:28.349384 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:28.349388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.349978 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.349979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:28.350124 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:28.351126 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:28.351138 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:28.352785 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:28.354861 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:28.354871 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:28.354873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:28.355259 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:28.355533 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:28.355534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:28.355534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:28.355696 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:28.356049 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:28.356057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:28.356260 (+ 203us) client_negotiation.cc:769] Sending connection context
0811 11:18:28.357058 (+ 798us) client_negotiation.cc:240] Negotiation successful
0811 11:18:28.357248 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260811 11:18:28.358186 10517 heartbeater.cc:347] Connected to a master server at 127.28.25.62:44173
I20260811 11:18:28.358284 10517 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:28.358525 10517 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
I20260811 11:18:28.358908 10116 ts_manager.cc:194] Registered new tserver with Master: 9fb6928b7c7b475ba75ac78814ef7d9c (127.28.25.2:43791)
I20260811 11:18:28.359462 10116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:43479
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:18:28.387501 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:48961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44173
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:28.503634 10523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:28.503954 10523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:28.504009 10523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:28.505877 10523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:28.505949 10523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:28.505995 10523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:18:28.508348 10523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.25.20:48961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:28.508701 10523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:28.508982 10523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:28.516320 10528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:28.516557 10529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:28.517612 10523 server_base.cc:1034] running on GCE node
W20260811 11:18:28.517820 10531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:28.518105 10523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:28.518393 10523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:28.521677 10523 hybrid_clock.cc:648] HybridClock initialized: now 1786447108521653 us; error 45 us; skew 500 ppm
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:28.525274 10523 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:18:28.525839 10523 webserver.cc:460] Webserver started at http://127.28.25.3:43859/ using document root <none> and password file <none>
I20260811 11:18:28.526096 10523 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:28.526151 10523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:28.526281 10523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:18:28.543727 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "7270fc7509624dfeb96ae06a29514888"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "2ac29d02b2948092b76ef4352bbd352e"
server_key_iv: "798cd4472df7f7719aa25b77ae6bb289"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:28.544203 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "7270fc7509624dfeb96ae06a29514888"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "2ac29d02b2948092b76ef4352bbd352e"
server_key_iv: "798cd4472df7f7719aa25b77ae6bb289"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:28.551544 10523 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260811 11:18:28.588781 10542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:28.589011 10523 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260811 11:18:28.589115 10523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "7270fc7509624dfeb96ae06a29514888"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "2ac29d02b2948092b76ef4352bbd352e"
server_key_iv: "798cd4472df7f7719aa25b77ae6bb289"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:28.589236 10523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:28.610014 10523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:28.612466 10523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:28.612623 10523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:28.612866 10523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:28.613226 10523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:28.613272 10523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:28.613301 10523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:28.613324 10523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:28.621635 10523 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:33693
I20260811 11:18:28.622269 10523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:18:28.622313 10670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:33693 every 8 connection(s)
I20260811 11:18:28.628463 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 10523
I20260811 11:18:28.628602 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:18:28.628791 28772 external_mini_cluster.cc:1428] Setting key 00e8b72898beaab89d44de1f01971f04
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:28.641012 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:28.623269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:34879 (local address 127.28.25.62:44173)
0811 11:18:28.623319 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:28.623321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:28.629101 (+ 5780us) server_negotiation.cc:407] Connection header received
0811 11:18:28.629918 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:28.629919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:28.629933 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:28.629993 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:28.630852 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.631686 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.633370 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.633670 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.633754 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:28.636683 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:28.636686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:28.636687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:28.636700 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:28.638683 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:28.639504 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:28.639506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:28.639507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:28.639595 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:28.640048 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:28.640049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:28.640050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:28.640230 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:28.640292 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:28.640767 (+ 475us) server_negotiation.cc:299] Negotiation successful
0811 11:18:28.640887 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:18:28.641160 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:28.623257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:44173 (local address 127.28.25.3:34879)
0811 11:18:28.628836 (+ 5579us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:28.628926 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:28.629779 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:28.630092 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:28.630097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:28.630401 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:28.630733 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:28.630741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.631814 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.631816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:28.633257 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:28.633261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:28.633802 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:28.633803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:28.633959 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:28.634705 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:28.634734 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:28.636534 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:28.638870 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:28.638882 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:28.638885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:28.639383 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:28.639691 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:28.639692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:28.639693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:28.639932 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:28.640331 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:28.640338 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:28.640562 (+ 224us) client_negotiation.cc:769] Sending connection context
0811 11:18:28.640682 (+ 120us) client_negotiation.cc:240] Negotiation successful
0811 11:18:28.640883 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5502,"thread_start_us":93,"threads_started":1}
I20260811 11:18:28.642148 10671 heartbeater.cc:347] Connected to a master server at 127.28.25.62:44173
I20260811 11:18:28.642287 10671 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:28.642599 10671 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
I20260811 11:18:28.643127 10116 ts_manager.cc:194] Registered new tserver with Master: 7270fc7509624dfeb96ae06a29514888 (127.28.25.3:33693)
I20260811 11:18:28.643674 10116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:34879
I20260811 11:18:28.651464 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:18:28.652419 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 10206
I20260811 11:18:28.668883 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 10360
I20260811 11:18:28.679409 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 10523
I20260811 11:18:28.700129 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 10080
I20260811 11:18:28.709791 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:44173
--webserver_interface=127.28.25.62
--webserver_port=45007
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:44173
--ranger_config_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:48961/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.25.20:48961/kms with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:28.820061 10674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:28.820325 10674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:28.820366 10674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:28.821601 10674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:18:28.821689 10674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:28.821715 10674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:28.821765 10674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:18:28.821790 10674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:18:28.823572 10674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.25.20:48961/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:44173
--ranger_config_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:28.823921 10674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:28.824165 10674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:28.830206 10683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:28.830173 10680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:28.830193 10681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:28.830330 10674 server_base.cc:1034] running on GCE node
I20260811 11:18:28.830760 10674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:28.831353 10674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:28.832535 10674 hybrid_clock.cc:648] HybridClock initialized: now 1786447108832513 us; error 47 us; skew 500 ppm
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:28.835479 10674 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:18:28.836014 10674 webserver.cc:460] Webserver started at http://127.28.25.62:45007/ using document root <none> and password file <none>
I20260811 11:18:28.836257 10674 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:28.836349 10674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 11:18:28 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:18:28.915050 10674 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:18:28.916128 10693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:28.916344 10674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:28.916447 10674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "f532cb2964a64bca9f0a3dade5e06a6a"
format_stamp: "Formatted at 2026-08-11 11:18:26 on dist-test-slave-2w3w"
server_key: "ef8cd2aaec28cc91badeca51f165907b"
server_key_iv: "c0ca8ab36f9da1f5e17560f906091fd7"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:28.916893 10674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:28.941906 10674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:28.944247 10674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:28.944408 10674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:28.949473 10674 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:44173
I20260811 11:18:28.949508 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:44173 every 8 connection(s)
I20260811 11:18:28.949944 10674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:18:28.951527 10759 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:18:28.952260 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: Bootstrap starting.
I20260811 11:18:28.958894 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 10674
I20260811 11:18:28.959379 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:48961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:38701
--local_ip_for_outbound_sockets=127.28.25.1
--tserver_master_addrs=127.28.25.62:44173
--webserver_port=37729
--webserver_interface=127.28.25.1
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.25.20:48961/kms with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:18:28.961234 10759 log.cc:824] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:28.963487 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:18:28.963851 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: Bootstrap complete.
I20260811 11:18:28.966528 10759 raft_consensus.cc:348] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } }
I20260811 11:18:28.966816 10759 raft_consensus.cc:729] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f532cb2964a64bca9f0a3dade5e06a6a, State: Initialized, Role: FOLLOWER
I20260811 11:18:28.967072 10759 consensus_queue.cc:260] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } }
I20260811 11:18:28.967180 10759 raft_consensus.cc:388] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:18:28.967237 10759 raft_consensus.cc:482] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:18:28.967294 10759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 1 FOLLOWER]: Advancing to term 2
I20260811 11:18:28.968398 10759 raft_consensus.cc:504] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } }
I20260811 11:18:28.968536 10759 leader_election.cc:302] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f532cb2964a64bca9f0a3dade5e06a6a; no voters:
I20260811 11:18:28.968748 10759 leader_election.cc:288] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 11:18:28.968842 10763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 2 FOLLOWER]: Leader election won for term 2
I20260811 11:18:28.969056 10763 raft_consensus.cc:686] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [term 2 LEADER]: Becoming Leader. State: Replica: f532cb2964a64bca9f0a3dade5e06a6a, State: Running, Role: LEADER
I20260811 11:18:28.969072 10759 sys_catalog.cc:564] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:18:28.969208 10763 consensus_queue.cc:237] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } }
I20260811 11:18:28.969573 10763 sys_catalog.cc:455] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f532cb2964a64bca9f0a3dade5e06a6a. Latest consensus state: current_term: 2 leader_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } } }
I20260811 11:18:28.969679 10763 sys_catalog.cc:458] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:28.970134 10764 sys_catalog.cc:455] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f532cb2964a64bca9f0a3dade5e06a6a" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44173 } } }
I20260811 11:18:28.970228 10764 sys_catalog.cc:458] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:28.970425 10774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:18:28.970794 10759 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 11:18:28.971112 10774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:18:28.971287 10774 catalog_manager.cc:1259] Loaded cluster ID: 2a9fa90f05514341aa95135a75895cd0
I20260811 11:18:28.971355 10774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:18:28.976378 10774 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:18:28.976971 10774 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: Loaded TSK: 0
I20260811 11:18:28.977370 10774 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 11:18:29.066129 10761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:29.066373 10761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:29.066418 10761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:29.067775 10761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:29.067855 10761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:29.067910 10761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:18:29.069924 10761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.25.20:48961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:38701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=37729
--tserver_master_addrs=127.28.25.62:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:29.070271 10761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:29.070554 10761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:29.076924 10805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:29.077013 10803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:29.077533 10802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:29.079197 10761 server_base.cc:1034] running on GCE node
I20260811 11:18:29.079474 10761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:29.079811 10761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:29.080983 10761 hybrid_clock.cc:648] HybridClock initialized: now 1786447109080970 us; error 30 us; skew 500 ppm
Aug 11 11:18:29 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:29.083941 10761 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:18:29.084666 10761 webserver.cc:460] Webserver started at http://127.28.25.1:37729/ using document root <none> and password file <none>
I20260811 11:18:29.084928 10761 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:29.085018 10761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 11:18:29 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:18:29.118875 10761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:18:29.119815 10815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:29.120012 10761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:18:29.120092 10761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "4cdb667263c44e6ca18904092b4bde7d"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "a4efb0c133fa17f8209269388f8579de"
server_key_iv: "7d579b7417e6fee806b889e1551d7a4e"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:29.120668 10761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:29.139375 10761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:29.141736 10761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:29.141940 10761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:29.142251 10761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:29.142685 10761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:29.142750 10761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:29.142792 10761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:29.142841 10761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:29.150491 10761 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:38701
I20260811 11:18:29.153007 10951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:38701 every 8 connection(s)
I20260811 11:18:29.153242 10761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:18:29.156833 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 10761
I20260811 11:18:29.157183 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:48961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:43791
--local_ip_for_outbound_sockets=127.28.25.2
--tserver_master_addrs=127.28.25.62:44173
--webserver_port=39787
--webserver_interface=127.28.25.2
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.25.20:48961/kms with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:18:29 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:29.174825 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:29.154536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:44173 (local address 127.28.25.1:38807)
0811 11:18:29.155348 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:29.155421 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:29.156099 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:29.157091 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:29.157096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:29.157390 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:29.157707 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:29.157714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.163128 (+ 5414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.163131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:29.164729 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:29.164738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.166025 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.166027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:29.166205 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:29.166937 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:29.166952 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:29.168847 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:29.172097 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:29.172113 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:29.172116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:29.172587 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:29.172970 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:29.172972 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:29.172972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:29.173164 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:29.173853 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:29.173862 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:29.174077 (+ 215us) client_negotiation.cc:769] Sending connection context
0811 11:18:29.174209 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 11:18:29.174451 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":102,"threads_started":1}
I20260811 11:18:29.175895 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:29.155018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:38807 (local address 127.28.25.62:44173)
0811 11:18:29.156391 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:29.156414 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:29.156460 (+ 46us) server_negotiation.cc:407] Connection header received
0811 11:18:29.156669 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:29.156677 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:29.156744 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:29.156897 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:29.157861 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.162791 (+ 4930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.164910 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.165710 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.165857 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:29.169059 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:29.169067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:29.169067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:29.169119 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:29.171947 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:29.172714 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:29.172719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:29.172719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:29.172876 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:29.173265 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:29.173267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:29.173267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:29.173716 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:29.173815 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:29.175440 (+ 1625us) server_negotiation.cc:299] Negotiation successful
0811 11:18:29.175614 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1295,"thread_start_us":108,"threads_started":1}
I20260811 11:18:29.177202 10952 heartbeater.cc:347] Connected to a master server at 127.28.25.62:44173
I20260811 11:18:29.177307 10952 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:29.177523 10952 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
W20260811 11:18:29.178467 10952 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:44173 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 11:18:29.178543 10952 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:29.178608 10952 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
I20260811 11:18:29.179046 10952 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:29.179124 10952 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
W20260811 11:18:29.179459 10952 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:18:29.277613 10956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:29.277943 10956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:29.277987 10956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:29.279929 10956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:29.279997 10956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:29.280061 10956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:18:29.282218 10956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.25.20:48961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:43791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=39787
--tserver_master_addrs=127.28.25.62:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:29.282503 10956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:29.282732 10956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:29.290127 10964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:29.290458 10967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:29.292446 10956 server_base.cc:1034] running on GCE node
W20260811 11:18:29.295199 10962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:29.295570 10956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:29.295982 10956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:29.297262 10956 hybrid_clock.cc:648] HybridClock initialized: now 1786447109297221 us; error 62 us; skew 500 ppm
Aug 11 11:18:29 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:29.301043 10956 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:18:29.301802 10956 webserver.cc:460] Webserver started at http://127.28.25.2:39787/ using document root <none> and password file <none>
I20260811 11:18:29.302165 10956 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:29.302332 10956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 11:18:29 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:18:29.331595 10956 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260811 11:18:29.332456 10975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:29.332676 10956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:29.332789 10956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "9fb6928b7c7b475ba75ac78814ef7d9c"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "6981928be1c38f624cd5f8424f141ec3"
server_key_iv: "49e766002477d2c50eebad259144415e"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:29.333216 10956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:29.357864 10956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:29.360541 10956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:29.360754 10956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:29.361058 10956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:29.361457 10956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:29.361514 10956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:29.361560 10956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:29.361588 10956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:29.369761 10956 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:43791
I20260811 11:18:29.369910 11111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:43791 every 8 connection(s)
I20260811 11:18:29.370312 10956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
Aug 11 11:18:29 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:29.378533 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 10956
I20260811 11:18:29.378909 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.25.20:48961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:33693
--local_ip_for_outbound_sockets=127.28.25.3
--tserver_master_addrs=127.28.25.62:44173
--webserver_port=43859
--webserver_interface=127.28.25.3
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.25.20:48961/kms with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:18:29.381516 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:29.371333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:38311 (local address 127.28.25.62:44173)
0811 11:18:29.371442 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:29.371444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:29.371686 (+ 242us) server_negotiation.cc:407] Connection header received
0811 11:18:29.372297 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:29.372299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:29.372313 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:29.372362 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:29.372980 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.373501 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.374602 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.374843 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.374936 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:29.377569 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:29.377571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:29.377572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:29.377582 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:29.379622 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:29.380273 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:29.380275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:29.380275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:29.380352 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:29.380709 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:29.380710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:29.380711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:29.380881 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:29.380967 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:29.381302 (+ 335us) server_negotiation.cc:299] Negotiation successful
0811 11:18:29.381420 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 11:18:29.381704 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:29.371350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:44173 (local address 127.28.25.2:38311)
0811 11:18:29.371526 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:29.371569 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:29.372180 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:29.372441 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:29.372445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:29.372658 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:29.372874 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:29.372879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.373611 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.373612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:29.374499 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:29.374501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.374996 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.374997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:29.375135 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:29.375741 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:29.375749 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:29.377419 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:29.379749 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:29.379762 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:29.379766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:29.380163 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:29.380438 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:29.380439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:29.380439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:29.380620 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:29.380974 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:29.380981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:29.381148 (+ 167us) client_negotiation.cc:769] Sending connection context
0811 11:18:29.381273 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 11:18:29.381446 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260811 11:18:29.382205 11112 heartbeater.cc:347] Connected to a master server at 127.28.25.62:44173
I20260811 11:18:29.382313 11112 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:29.382545 11112 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
W20260811 11:18:29.383285 11112 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:44173 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 11:18:29.383378 11112 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:29.383458 11112 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
I20260811 11:18:29.383761 11112 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:29.383839 11112 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
W20260811 11:18:29.384130 11112 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:18:29.510087 11115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:29.510412 11115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:29.510454 11115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:29.512405 11115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:29.512475 11115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:29.512519 11115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:18:29.514981 11115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.25.20:48961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:33693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=43859
--tserver_master_addrs=127.28.25.62:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:29.515333 11115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:29.515647 11115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:29.522552 11124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:29.522552 11121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:29.523533 11122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:29.523828 11115 server_base.cc:1034] running on GCE node
I20260811 11:18:29.524068 11115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:29.524406 11115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:29.525578 11115 hybrid_clock.cc:648] HybridClock initialized: now 1786447109525545 us; error 47 us; skew 500 ppm
Aug 11 11:18:29 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:29.528913 11115 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:18:29.529495 11115 webserver.cc:460] Webserver started at http://127.28.25.3:43859/ using document root <none> and password file <none>
I20260811 11:18:29.529793 11115 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:29.529867 11115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 11:18:29 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:18:29.546008 11115 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:18:29.547132 11135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:29.547304 11115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:29.547389 11115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "7270fc7509624dfeb96ae06a29514888"
format_stamp: "Formatted at 2026-08-11 11:18:28 on dist-test-slave-2w3w"
server_key: "2ac29d02b2948092b76ef4352bbd352e"
server_key_iv: "798cd4472df7f7719aa25b77ae6bb289"
server_key_version: "kuduclusterkey@0"
I20260811 11:18:29.547866 11115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:29.576460 11115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:29.579209 11115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:29.579397 11115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:29.579660 11115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:29.580085 11115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:29.580142 11115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:29.580174 11115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:29.580192 11115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:29.588364 11115 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:33693
I20260811 11:18:29.588999 11115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:18:29.589510 11265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:33693 every 8 connection(s)
Aug 11 11:18:29 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:29.599080 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 11115
I20260811 11:18:29.602099 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:29.590132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:49911 (local address 127.28.25.62:44173)
0811 11:18:29.590193 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:29.590196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:29.590440 (+ 244us) server_negotiation.cc:407] Connection header received
0811 11:18:29.591230 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:29.591232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:29.591257 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:29.591290 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:29.592206 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.593027 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.594701 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.595075 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.595126 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:29.597905 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:29.597908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:29.597908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:29.597918 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:29.600065 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:29.600846 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:29.600848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:29.600849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:29.600914 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:29.601211 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:29.601212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:29.601213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:29.601395 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:29.601451 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:29.601896 (+ 445us) server_negotiation.cc:299] Negotiation successful
0811 11:18:29.601992 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 11:18:29.602830 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:29.590129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:44173 (local address 127.28.25.3:49911)
0811 11:18:29.590301 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:29.590347 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:29.591084 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:29.591498 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:29.591502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:29.591778 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:29.592082 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:29.592089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.593164 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.593165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:29.594581 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:29.594584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.595250 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.595251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:29.595405 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:29.596142 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:29.596154 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:29.597748 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:29.600353 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:29.600367 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:29.600371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:29.600741 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:29.600987 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:29.600988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:29.600988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:29.601135 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:29.601511 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:29.601520 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:29.601728 (+ 208us) client_negotiation.cc:769] Sending connection context
0811 11:18:29.602377 (+ 649us) client_negotiation.cc:240] Negotiation successful
0811 11:18:29.602579 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260811 11:18:29.603494 11266 heartbeater.cc:347] Connected to a master server at 127.28.25.62:44173
I20260811 11:18:29.603595 11266 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:29.603801 11266 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
W20260811 11:18:29.604466 11266 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:44173 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 11:18:29.604537 11266 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:29.604596 11266 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
I20260811 11:18:29.604916 11266 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:29.604987 11266 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
W20260811 11:18:29.605291 11266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 11:18:29.610718 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:29.600194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.28.25.62:44173)
0811 11:18:29.601215 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:29.601218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:29.601483 (+ 265us) server_negotiation.cc:407] Connection header received
0811 11:18:29.601508 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:29.601509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:29.601524 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:29.601579 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:29.602662 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.603735 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.604998 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:29.605320 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:29.605436 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:29.607240 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:29.607242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:29.607243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:29.607252 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:29.609247 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:29.609715 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:29.609717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:29.609718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:29.609796 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:29.610050 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:29.610051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:29.610051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:29.610214 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:29.610443 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:29.610495 (+ 52us) server_negotiation.cc:299] Negotiation successful
0811 11:18:29.610597 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":108,"threads_started":1}
Aug 11 11:18:30 dist-test-slave-2w3w krb5kdc[9568](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786447110, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:30.179658 10952 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:30.179827 10952 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
I20260811 11:18:30.180322 10714 ts_manager.cc:194] Registered new tserver with Master: 4cdb667263c44e6ca18904092b4bde7d (127.28.25.1:38701)
I20260811 11:18:30.181298 10714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:38807
I20260811 11:18:30.384349 11112 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:30.384573 11112 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
I20260811 11:18:30.385089 10714 ts_manager.cc:194] Registered new tserver with Master: 9fb6928b7c7b475ba75ac78814ef7d9c (127.28.25.2:43791)
I20260811 11:18:30.385555 10714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:38311
Aug 11 11:18:30 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447110, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
I20260811 11:18:30.605460 11266 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:30.605638 11266 heartbeater.cc:511] Master 127.28.25.62:44173 requested a full tablet report, sending...
I20260811 11:18:30.606012 10714 ts_manager.cc:194] Registered new tserver with Master: 7270fc7509624dfeb96ae06a29514888 (127.28.25.3:33693)
I20260811 11:18:30.606545 10714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:49911
I20260811 11:18:30.615852 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:18:30.633071 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.616946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.28.25.62:44173)
0811 11:18:30.623357 (+ 6411us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:30.623360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:30.623373 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:18:30.623407 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:30.623408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:30.623426 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:30.623949 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:30.625155 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.625930 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.627037 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.627332 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.627403 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.629131 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:30.629133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:30.629134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:30.629144 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:30.631116 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:30.631622 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:30.631625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:30.631625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:30.631704 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:30.632006 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:30.632007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:30.632008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:30.632174 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:30.632277 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:30.632854 (+ 577us) server_negotiation.cc:299] Negotiation successful
0811 11:18:30.632955 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6108,"thread_start_us":70,"threads_started":1}
I20260811 11:18:30.635775 10714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:18:30.637811 10714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:18:30.648191 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.642899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:38701 (local address 127.0.0.1:55490)
0811 11:18:30.643658 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:30.643667 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:30.643711 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:30.644008 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:30.644012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:30.644014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:30.644089 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:30.644091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.645396 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.645398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:30.646730 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:30.646734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.647858 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.647860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:30.647948 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.647964 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 11:18:30.648039 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 11:18:30.648098 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":106,"threads_started":1}
I20260811 11:18:30.648571 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.642664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.28.25.1:38701)
0811 11:18:30.643609 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:30.643615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:30.643763 (+ 148us) server_negotiation.cc:407] Connection header received
0811 11:18:30.643802 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:30.643804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:30.643821 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:30.643898 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:30.644177 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.645275 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.646837 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.647639 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.647731 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.647818 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:30.648446 (+ 628us) server_negotiation.cc:299] Negotiation successful
0811 11:18:30.648491 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":164,"threads_started":1}
I20260811 11:18:30.649263 10848 tablet_service.cc:1467] Processing CreateTablet for tablet 83f0f8e981ee48f1be491837a29f9726 (DEFAULT_TABLE table=test-table [id=49a6fd8eacf546eda36f272c5576d06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:30.649693 10848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f0f8e981ee48f1be491837a29f9726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:30.654037 11305 tablet_bootstrap.cc:492] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d: Bootstrap starting.
I20260811 11:18:30.654807 11305 tablet_bootstrap.cc:654] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:30.655171 11305 log.cc:824] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:30.656026 11305 tablet_bootstrap.cc:492] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d: No bootstrap required, opened a new log
I20260811 11:18:30.656111 11305 ts_tablet_manager.cc:1397] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 11:18:30.657030 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.642532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:43791 (local address 127.0.0.1:34256)
0811 11:18:30.643169 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:30.643219 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:30.643296 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:30.648448 (+ 5152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:30.648450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:30.648452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:30.648495 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:30.648496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.650918 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.650919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:30.652663 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:30.652667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.656788 (+ 4121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.656789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:30.656859 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.656865 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:18:30.656896 (+ 31us) client_negotiation.cc:240] Negotiation successful
0811 11:18:30.656947 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":1024,"threads_started":1}
I20260811 11:18:30.657500 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.643176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.28.25.2:43791)
0811 11:18:30.648199 (+ 5023us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:30.648204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:30.648227 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:18:30.648263 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:30.648265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:30.648281 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:30.648359 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:30.648564 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.649550 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.652783 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.653826 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.657241 (+ 3415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.657338 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:30.657369 (+ 31us) server_negotiation.cc:299] Negotiation successful
0811 11:18:30.657419 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4977,"thread_start_us":64,"threads_started":1}
I20260811 11:18:30.658068 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.642821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:33693 (local address 127.0.0.1:33590)
0811 11:18:30.643433 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:30.643442 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:30.643498 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:30.649918 (+ 6420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:30.649920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:30.649922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:30.649964 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:30.649965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.654614 (+ 4649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.654615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:30.655813 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:30.655817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.657847 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.657848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:30.657918 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.657922 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:18:30.657954 (+ 32us) client_negotiation.cc:240] Negotiation successful
0811 11:18:30.657995 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":75,"threads_started":1}
I20260811 11:18:30.658218 11036 tablet_service.cc:1467] Processing CreateTablet for tablet 83f0f8e981ee48f1be491837a29f9726 (DEFAULT_TABLE table=test-table [id=49a6fd8eacf546eda36f272c5576d06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:30.658619 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f0f8e981ee48f1be491837a29f9726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:30.658574 11305 raft_consensus.cc:348] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 } } peers { permanent_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43791 } } peers { permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 } }
I20260811 11:18:30.658797 11305 raft_consensus.cc:374] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:30.658849 11305 raft_consensus.cc:729] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdb667263c44e6ca18904092b4bde7d, State: Initialized, Role: FOLLOWER
I20260811 11:18:30.658878 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.643322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.28.25.3:33693)
0811 11:18:30.649704 (+ 6382us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:30.649709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:30.649720 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:18:30.649754 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:30.649756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:30.649772 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:30.649844 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:30.650031 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.654524 (+ 4493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.655937 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.656652 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.658615 (+ 1963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.658719 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:30.658752 (+ 33us) server_negotiation.cc:299] Negotiation successful
0811 11:18:30.658806 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6339,"thread_start_us":60,"threads_started":1}
I20260811 11:18:30.659037 11305 consensus_queue.cc:260] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 } } peers { permanent_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43791 } } peers { permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 } }
I20260811 11:18:30.659329 11305 ts_tablet_manager.cc:1428] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 11:18:30.659469 10952 heartbeater.cc:503] Master 127.28.25.62:44173 was elected leader, sending a full tablet report...
I20260811 11:18:30.659622 11170 tablet_service.cc:1467] Processing CreateTablet for tablet 83f0f8e981ee48f1be491837a29f9726 (DEFAULT_TABLE table=test-table [id=49a6fd8eacf546eda36f272c5576d06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:30.659974 11170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f0f8e981ee48f1be491837a29f9726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:30.662472 11309 tablet_bootstrap.cc:492] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c: Bootstrap starting.
I20260811 11:18:30.663331 11309 tablet_bootstrap.cc:654] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:30.663667 11309 log.cc:824] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:30.664472 11309 tablet_bootstrap.cc:492] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c: No bootstrap required, opened a new log
I20260811 11:18:30.664453 11311 tablet_bootstrap.cc:492] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: Bootstrap starting.
I20260811 11:18:30.664548 11309 ts_tablet_manager.cc:1397] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 11:18:30.665127 11311 tablet_bootstrap.cc:654] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:30.665463 11311 log.cc:824] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:30.666347 11311 tablet_bootstrap.cc:492] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: No bootstrap required, opened a new log
I20260811 11:18:30.666424 11311 ts_tablet_manager.cc:1397] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:18:30.667057 11309 raft_consensus.cc:348] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 } } peers { permanent_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43791 } } peers { permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 } }
I20260811 11:18:30.667259 11309 raft_consensus.cc:374] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:30.667320 11309 raft_consensus.cc:729] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb6928b7c7b475ba75ac78814ef7d9c, State: Initialized, Role: FOLLOWER
I20260811 11:18:30.667500 11309 consensus_queue.cc:260] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 } } peers { permanent_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43791 } } peers { permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 } }
I20260811 11:18:30.667798 11309 ts_tablet_manager.cc:1428] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260811 11:18:30.667937 11112 heartbeater.cc:503] Master 127.28.25.62:44173 was elected leader, sending a full tablet report...
I20260811 11:18:30.669340 11311 raft_consensus.cc:348] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 } } peers { permanent_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43791 } } peers { permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 } }
I20260811 11:18:30.669529 11311 raft_consensus.cc:374] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:30.669562 11311 raft_consensus.cc:729] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7270fc7509624dfeb96ae06a29514888, State: Initialized, Role: FOLLOWER
I20260811 11:18:30.669708 11311 consensus_queue.cc:260] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 } } peers { permanent_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43791 } } peers { permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 } }
I20260811 11:18:30.670132 11311 ts_tablet_manager.cc:1428] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 11:18:30.670372 11266 heartbeater.cc:503] Master 127.28.25.62:44173 was elected leader, sending a full tablet report...
W20260811 11:18:30.840828 11267 tablet.cc:2369] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 11:18:30 dist-test-slave-2w3w krb5kdc[9568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447110, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for HTTP/127.28.25.20@KRBTEST.COM
W20260811 11:18:30.872175 11113 tablet.cc:2369] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:18:30.874234 11313 raft_consensus.cc:482] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:30.874404 11313 raft_consensus.cc:504] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 } } peers { permanent_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43791 } } peers { permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 } }
I20260811 11:18:30.874832 11313 leader_election.cc:288] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cdb667263c44e6ca18904092b4bde7d (127.28.25.1:38701), 7270fc7509624dfeb96ae06a29514888 (127.28.25.3:33693)
I20260811 11:18:30.879890 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.875385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:58775 (local address 127.28.25.1:38701)
0811 11:18:30.875443 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:30.875445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:30.876175 (+ 730us) server_negotiation.cc:407] Connection header received
0811 11:18:30.876292 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:30.876294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:30.876309 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:30.876341 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:30.876614 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.877311 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.878806 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.879436 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.879481 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.879494 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:30.879754 (+ 260us) server_negotiation.cc:299] Negotiation successful
0811 11:18:30.879807 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 11:18:30.880287 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.875123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:38701 (local address 127.28.25.2:58775)
0811 11:18:30.876094 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:30.876107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:30.876210 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:30.876424 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:30.876424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:30.876426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:30.876528 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:30.876530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.877440 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.877441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:30.878711 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:30.878713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.879552 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.879553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:30.879652 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.879655 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:18:30.880155 (+ 500us) client_negotiation.cc:240] Negotiation successful
0811 11:18:30.880206 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"mutex_wait_us":17,"thread_start_us":123,"threads_started":1}
I20260811 11:18:30.880676 10897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f0f8e981ee48f1be491837a29f9726" candidate_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdb667263c44e6ca18904092b4bde7d" is_pre_election: true
I20260811 11:18:30.880867 10897 raft_consensus.cc:2454] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb6928b7c7b475ba75ac78814ef7d9c in term 0.
I20260811 11:18:30.881109 10977 leader_election.cc:302] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cdb667263c44e6ca18904092b4bde7d, 9fb6928b7c7b475ba75ac78814ef7d9c; no voters:
I20260811 11:18:30.881244 11313 raft_consensus.cc:2781] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:18:30.881282 11313 raft_consensus.cc:482] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:18:30.881305 11313 raft_consensus.cc:3037] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:30.881783 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.875894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:37435 (local address 127.28.25.3:33693)
0811 11:18:30.875951 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:30.875953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:30.878147 (+ 2194us) server_negotiation.cc:407] Connection header received
0811 11:18:30.878250 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:30.878251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:30.878264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:30.878291 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:30.878492 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.879199 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.880676 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.881291 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.881343 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.881354 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:30.881661 (+ 307us) server_negotiation.cc:299] Negotiation successful
0811 11:18:30.881713 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 11:18:30.882107 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.875673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:33693 (local address 127.28.25.2:37435)
0811 11:18:30.878063 (+ 2390us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:30.878077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:30.878177 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:30.878350 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:30.878351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:30.878352 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:30.878417 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:30.878419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.879300 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.879301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:30.880562 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:30.880566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.881500 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.881501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:30.881572 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.881576 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:18:30.881956 (+ 380us) client_negotiation.cc:240] Negotiation successful
0811 11:18:30.882004 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2369,"thread_start_us":77,"threads_started":1}
I20260811 11:18:30.882296 11313 raft_consensus.cc:504] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 } } peers { permanent_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43791 } } peers { permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 } }
I20260811 11:18:30.882467 11313 leader_election.cc:288] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [CANDIDATE]: Term 1 election: Requested vote from peers 4cdb667263c44e6ca18904092b4bde7d (127.28.25.1:38701), 7270fc7509624dfeb96ae06a29514888 (127.28.25.3:33693)
I20260811 11:18:30.882543 11216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f0f8e981ee48f1be491837a29f9726" candidate_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7270fc7509624dfeb96ae06a29514888" is_pre_election: true
I20260811 11:18:30.882678 11215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f0f8e981ee48f1be491837a29f9726" candidate_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7270fc7509624dfeb96ae06a29514888"
I20260811 11:18:30.882736 11216 raft_consensus.cc:2454] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb6928b7c7b475ba75ac78814ef7d9c in term 0.
I20260811 11:18:30.883347 10897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f0f8e981ee48f1be491837a29f9726" candidate_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdb667263c44e6ca18904092b4bde7d"
I20260811 11:18:30.883456 10897 raft_consensus.cc:3037] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:30.884608 10897 raft_consensus.cc:2454] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb6928b7c7b475ba75ac78814ef7d9c in term 1.
I20260811 11:18:30.884852 10977 leader_election.cc:302] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cdb667263c44e6ca18904092b4bde7d, 9fb6928b7c7b475ba75ac78814ef7d9c; no voters: 7270fc7509624dfeb96ae06a29514888
I20260811 11:18:30.885037 11313 raft_consensus.cc:2781] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:18:30.885236 11313 raft_consensus.cc:686] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 1 LEADER]: Becoming Leader. State: Replica: 9fb6928b7c7b475ba75ac78814ef7d9c, State: Running, Role: LEADER
I20260811 11:18:30.885355 11313 consensus_queue.cc:237] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 } } peers { permanent_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43791 } } peers { permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 } }
I20260811 11:18:30.886348 10714 catalog_manager.cc:5495] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fb6928b7c7b475ba75ac78814ef7d9c (127.28.25.2). New cstate: current_term: 1 leader_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fb6928b7c7b475ba75ac78814ef7d9c" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 } health_report { overall_health: UNKNOWN } } }
W20260811 11:18:30.905227 10953 tablet.cc:2369] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:18:30.933960 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.929394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34262 (local address 127.28.25.2:43791)
0811 11:18:30.929471 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:30.929474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:30.929484 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:18:30.929531 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:30.929532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:30.929559 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:30.929645 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:18:30.929839 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.930613 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.931589 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:30.931864 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:30.931939 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:30.932152 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:18:30.932300 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:18:30.932395 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:30.933835 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0811 11:18:30.933871 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:18:30.936061 11215 raft_consensus.cc:3037] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:30.936137 10897 raft_consensus.cc:1261] T 83f0f8e981ee48f1be491837a29f9726 P 4cdb667263c44e6ca18904092b4bde7d [term 1 FOLLOWER]: Refusing update from remote peer 9fb6928b7c7b475ba75ac78814ef7d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:18:30.936513 11320 consensus_queue.cc:1035] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cdb667263c44e6ca18904092b4bde7d" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:18:30.937204 11215 raft_consensus.cc:1261] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888 [term 1 FOLLOWER]: Refusing update from remote peer 9fb6928b7c7b475ba75ac78814ef7d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:18:30.941249 11320 consensus_queue.cc:1035] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7270fc7509624dfeb96ae06a29514888" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:18:30.944514 11325 mvcc.cc:205] Tried to move back new op lower bound from 7317287366391427072 to 7317287366187261952. Current Snapshot: MvccSnapshot[applied={T|T < 7317287366391427072 or (T in {7317287366391427072})}]
I20260811 11:18:30.946513 10714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:18:30.946647 10714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:18:30.951421 10714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f532cb2964a64bca9f0a3dade5e06a6a: Sending DeleteTablet for 3 replicas of tablet 83f0f8e981ee48f1be491837a29f9726
I20260811 11:18:30.952195 10848 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f0f8e981ee48f1be491837a29f9726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:30 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:55490
I20260811 11:18:30.952549 11170 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f0f8e981ee48f1be491837a29f9726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:30 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:33590
I20260811 11:18:30.953096 11036 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f0f8e981ee48f1be491837a29f9726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:30 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:34256
I20260811 11:18:30.954083 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 10761
I20260811 11:18:30.959139 11335 tablet_replica.cc:330] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: stopping tablet replica
I20260811 11:18:30.959446 11335 raft_consensus.cc:2229] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:18:30.959699 11335 raft_consensus.cc:2258] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:18:30.960609 11336 tablet_replica.cc:330] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c: stopping tablet replica
I20260811 11:18:30.961021 11335 ts_tablet_manager.cc:1905] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:18:30.961205 11336 raft_consensus.cc:2229] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:18:30.963276 11336 raft_consensus.cc:2258] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:18:30.963660 11335 ts_tablet_manager.cc:1918] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:18:30.963802 11335 log.cc:1196] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/83f0f8e981ee48f1be491837a29f9726
I20260811 11:18:30.964228 11335 ts_tablet_manager.cc:1939] T 83f0f8e981ee48f1be491837a29f9726 P 7270fc7509624dfeb96ae06a29514888: Deleting consensus metadata
I20260811 11:18:30.965431 11336 ts_tablet_manager.cc:1905] T 83f0f8e981ee48f1be491837a29f9726 P 9fb6928b7c7b475ba75ac78814ef7d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260811 11:18:30.966806 10694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:38701 (error 108)
I20260811 11:18:30.967680 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 10956
I20260811 11:18:30.976863 10697 catalog_manager.cc:4841] TS 7270fc7509624dfeb96ae06a29514888 (127.28.25.3:33693): tablet 83f0f8e981ee48f1be491837a29f9726 (table test-table [id=49a6fd8eacf546eda36f272c5576d06a]) successfully deleted
I20260811 11:18:30.977841 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.977633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:38701 (local address 127.0.0.1:55492)
0811 11:18:30.977721 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:30.977796 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:38701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260811 11:18:30.978330 10694 catalog_manager.cc:4584] TS 4cdb667263c44e6ca18904092b4bde7d (127.28.25.1:38701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83f0f8e981ee48f1be491837a29f9726: Network error: Client connection negotiation failed: client connection to 127.28.25.1:38701: connect: Connection refused (error 111)
I20260811 11:18:30.984365 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 11115
I20260811 11:18:30.988065 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:30.987902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:43791 (local address 127.0.0.1:34264)
0811 11:18:30.987977 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:30.988014 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.2:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 11:18:30.994488 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 10674
I20260811 11:18:31.004333 28772 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 11:18:31.350168 28772 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 11:18:31.350297 28772 mini_ranger.cc:66] Stopping Ranger...
I20260811 11:18:31.706496 28772 mini_ranger.cc:68] Stopped Ranger
2026-08-11 11:18:31.706 UTC [9602] LOG: received smart shutdown request
2026-08-11 11:18:31.713 UTC [9602] LOG: background worker "logical replication launcher" (PID 9613) exited with exit code 1
2026-08-11 11:18:31.713 UTC [9607] LOG: shutting down
2026-08-11 11:18:31.965 UTC [9602] LOG: database system is shut down
2026-08-11T11:18:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57209 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:18:32 dist-test-slave-2w3w krb5kdc[11348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:18:32 dist-test-slave-2w3w krb5kdc[11348](info): set up 2 sockets
Aug 11 11:18:32 dist-test-slave-2w3w krb5kdc[11348](info): commencing operation
krb5kdc: starting...
W20260811 11:18:33.239405 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:18:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:18:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:18:33.383347 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:46705
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:46705,127.28.25.61:45455,127.28.25.60:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:33.472391 11362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:33.472683 11362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:33.472728 11362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:33.474103 11362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:18:33.474200 11362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:33.474224 11362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:33.474247 11362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:18:33.474278 11362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:18:33.476157 11362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:46705,127.28.25.61:45455,127.28.25.60:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:46705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:33.476507 11362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:33.476769 11362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:33.482239 11368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:33.482383 11371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:33.482537 11362 server_base.cc:1034] running on GCE node
W20260811 11:18:33.482239 11369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:33.483029 11362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:33.483377 11362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:33.484566 11362 hybrid_clock.cc:648] HybridClock initialized: now 1786447113484558 us; error 46 us; skew 500 ppm
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:33.487633 11362 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:18:33.488250 11362 webserver.cc:460] Webserver started at http://127.28.25.62:35409/ using document root <none> and password file <none>
I20260811 11:18:33.488524 11362 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:33.488600 11362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:33.488770 11362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:33.489989 11362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "ea477beeee2a4f96b6239ed8e63ad92c"
format_stamp: "Formatted at 2026-08-11 11:18:33 on dist-test-slave-2w3w"
server_key: "c7182f4ad6075c1dfc4660fe7898eb81"
server_key_iv: "40b569ca367e74c687a3e72abc36343c"
server_key_version: "encryptionkey@0"
I20260811 11:18:33.490399 11362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "ea477beeee2a4f96b6239ed8e63ad92c"
format_stamp: "Formatted at 2026-08-11 11:18:33 on dist-test-slave-2w3w"
server_key: "c7182f4ad6075c1dfc4660fe7898eb81"
server_key_iv: "40b569ca367e74c687a3e72abc36343c"
server_key_version: "encryptionkey@0"
I20260811 11:18:33.492137 11362 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260811 11:18:33.493079 11379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:33.493345 11362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:18:33.493476 11362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "ea477beeee2a4f96b6239ed8e63ad92c"
format_stamp: "Formatted at 2026-08-11 11:18:33 on dist-test-slave-2w3w"
server_key: "c7182f4ad6075c1dfc4660fe7898eb81"
server_key_iv: "40b569ca367e74c687a3e72abc36343c"
server_key_version: "encryptionkey@0"
I20260811 11:18:33.493569 11362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:33.518023 11362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:33.520926 11362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:33.521134 11362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:33.525928 11362 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:46705
I20260811 11:18:33.526002 11437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:46705 every 8 connection(s)
I20260811 11:18:33.526448 11362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:18:33.527647 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:33.529340 11438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:18:33.530233 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 11362
I20260811 11:18:33.530354 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:18:33.530581 28772 external_mini_cluster.cc:1428] Setting key ed320560fc2d7637d66c4ad452b2c1ab
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.25.61@KRBTEST.COM; defaulting to no policy
I20260811 11:18:33.541401 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.530421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44804 (local address 127.28.25.62:46705)
0811 11:18:33.530579 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.530584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.530673 (+ 89us) server_negotiation.cc:407] Connection header received
0811 11:18:33.531363 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.531366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.531392 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.531504 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.532223 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.533382 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.534460 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.534819 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.535032 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.537443 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.537451 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.537451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.537459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.539469 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.539951 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.539955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.539955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.540053 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.540360 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.540362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.540362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.540611 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.540715 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.540956 (+ 241us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.541117 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260811 11:18:33.541405 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.530147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46705 (local address 127.0.0.1:44804)
0811 11:18:33.530504 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.530562 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:33.531178 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:33.531540 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:33.531543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:33.531914 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:33.532103 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.532108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.533489 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.533490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.534351 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.534355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.534987 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.534988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.535066 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.535743 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:33.535753 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:33.537315 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:33.539575 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.539584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.539586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.539843 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.540135 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.540136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.540136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.540247 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.540692 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:33.540698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:33.540785 (+ 87us) client_negotiation.cc:769] Sending connection context
0811 11:18:33.540914 (+ 129us) client_negotiation.cc:240] Negotiation successful
0811 11:18:33.541191 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":102,"thread_start_us":111,"threads_started":1}
I20260811 11:18:33.542275 11438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:18:33.542901 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.542718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:45455 (local address 127.0.0.1:60154)
0811 11:18:33.542791 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.542867 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:45455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 11:18:33.543123 11383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.25.61:45455: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.61@KRBTEST.COM" created.
I20260811 11:18:33.543845 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.543699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:45455 (local address 127.0.0.1:60164)
0811 11:18:33.543774 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.543811 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:45455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 11:18:33.544045 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.61:45455: Network error: Client connection negotiation failed: client connection to 127.28.25.61:45455: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab.
Entry for principal kudu/127.28.25.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab.
I20260811 11:18:33.563469 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.61:45455
--webserver_interface=127.28.25.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab
--principal=kudu/127.28.25.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:46705,127.28.25.61:45455,127.28.25.60:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:18:33.593410 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } attempt: 1
I20260811 11:18:33.594012 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.593822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:45455 (local address 127.0.0.1:60168)
0811 11:18:33.593910 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.593969 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:45455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260811 11:18:33.594551 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.594417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:45455 (local address 127.0.0.1:60184)
0811 11:18:33.594480 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.594516 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:45455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 11:18:33.594738 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.61:45455: Network error: Client connection negotiation failed: client connection to 127.28.25.61:45455: connect: Connection refused (error 111)
W20260811 11:18:33.654546 11445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:33.654824 11445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:33.654852 11445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:33.656168 11445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:18:33.656229 11445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:33.656244 11445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:33.656257 11445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:18:33.656270 11445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:18:33.658047 11445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:46705,127.28.25.61:45455,127.28.25.60:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab
--principal=kudu/127.28.25.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.61:45455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.25.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:33.658347 11445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:33.658625 11445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:18:33.662935 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } attempt: 2
I20260811 11:18:33.663528 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.663340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:45455 (local address 127.0.0.1:60194)
0811 11:18:33.663414 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.663462 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:45455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260811 11:18:33.664124 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.663971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:45455 (local address 127.0.0.1:60200)
0811 11:18:33.664041 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.664087 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:45455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 11:18:33.663998 11453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:33.664012 11452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:33.664011 11456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:33.664327 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.61:45455: Network error: Client connection negotiation failed: client connection to 127.28.25.61:45455: connect: Connection refused (error 111)
I20260811 11:18:33.664649 11445 server_base.cc:1034] running on GCE node
I20260811 11:18:33.665012 11445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:33.665385 11445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:33.666572 11445 hybrid_clock.cc:648] HybridClock initialized: now 1786447113666544 us; error 41 us; skew 500 ppm
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:33.669700 11445 init.cc:377] Logged in from keytab as kudu/127.28.25.61@KRBTEST.COM (short username kudu)
I20260811 11:18:33.670348 11445 webserver.cc:460] Webserver started at http://127.28.25.61:34483/ using document root <none> and password file <none>
I20260811 11:18:33.670598 11445 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:33.670670 11445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:33.670835 11445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:33.672093 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/data/instance:
uuid: "7b6c48c1ea854cde8f233f0555147c18"
format_stamp: "Formatted at 2026-08-11 11:18:33 on dist-test-slave-2w3w"
server_key: "d6c6043605a84ae7e8e75b2c1b29f98c"
server_key_iv: "2a38e7b4dddf1a4e46c050de8c86a57c"
server_key_version: "encryptionkey@0"
I20260811 11:18:33.672514 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/wal/instance:
uuid: "7b6c48c1ea854cde8f233f0555147c18"
format_stamp: "Formatted at 2026-08-11 11:18:33 on dist-test-slave-2w3w"
server_key: "d6c6043605a84ae7e8e75b2c1b29f98c"
server_key_iv: "2a38e7b4dddf1a4e46c050de8c86a57c"
server_key_version: "encryptionkey@0"
I20260811 11:18:33.674175 11445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:18:33.675382 11463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:33.675678 11445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:18:33.675837 11445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/wal
uuid: "7b6c48c1ea854cde8f233f0555147c18"
format_stamp: "Formatted at 2026-08-11 11:18:33 on dist-test-slave-2w3w"
server_key: "d6c6043605a84ae7e8e75b2c1b29f98c"
server_key_iv: "2a38e7b4dddf1a4e46c050de8c86a57c"
server_key_version: "encryptionkey@0"
I20260811 11:18:33.675942 11445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:33.704811 11445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:33.707613 11445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:33.707823 11445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:33.712517 11445 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.61:45455
I20260811 11:18:33.712558 11523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.61:45455 every 8 connection(s)
I20260811 11:18:33.712973 11445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
I20260811 11:18:33.714098 11524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:33.716025 11524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:18:33.721095 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 11445
I20260811 11:18:33.721211 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-1/wal/instance
I20260811 11:18:33.721475 28772 external_mini_cluster.cc:1428] Setting key fcec2e1c2f8260cdc2cd71063103d3a6
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:33.727247 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.716879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.28.25.62:46705)
0811 11:18:33.716955 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.716958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.717124 (+ 166us) server_negotiation.cc:407] Connection header received
0811 11:18:33.717641 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.717642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.717657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.717730 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.718537 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.719186 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.720348 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.720568 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.720640 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.723374 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.723377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.723377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.723388 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.725262 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.725787 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.725789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.725789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.725864 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.726201 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.726203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.726203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.726391 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.726475 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.726940 (+ 465us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.727083 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 11:18:33.727360 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.716780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46705 (local address 127.0.0.1:44808)
0811 11:18:33.716984 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.717031 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:33.717530 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:33.717813 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:33.717817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:33.718096 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:33.718362 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.718367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.719287 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.719288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.720170 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.720173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.720684 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.720685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.720813 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.721649 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:33.721659 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:33.723208 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:33.725412 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.725422 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.725424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.725693 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.725968 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.725969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.725970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.726108 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.726509 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:33.726519 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:33.726712 (+ 193us) client_negotiation.cc:769] Sending connection context
0811 11:18:33.726884 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 11:18:33.727117 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":111,"threads_started":1}
I20260811 11:18:33.728192 11524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.25.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.60@KRBTEST.COM" created.
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
I20260811 11:18:33.739333 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.728544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:45455 (local address 127.0.0.1:60206)
0811 11:18:33.728610 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.728623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:33.728663 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:33.729053 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:33.729054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:33.729241 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:33.729319 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.729321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.730316 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.730317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.730834 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.730836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.731446 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.731447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.731521 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.732078 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:33.732083 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:33.735374 (+ 3291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:33.737789 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.737790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.737791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.738062 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.738394 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.738395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.738395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.738464 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.738875 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:33.738876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:33.738923 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 11:18:33.739041 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 11:18:33.739170 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 11:18:33.739411 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.728626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.28.25.61:45455)
0811 11:18:33.728862 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.728867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.728890 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:18:33.728910 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.728912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.728924 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.729050 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.729411 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.730163 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.731009 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.731352 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.731454 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.735497 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.735515 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.735515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.735526 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.737670 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.738188 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.738191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.738192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.738285 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.738539 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.738541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.738541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.738783 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.738893 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.739079 (+ 186us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.739279 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":195,"thread_start_us":128,"threads_started":1}
I20260811 11:18:33.740350 11524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:18:33.740813 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.740653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51606)
0811 11:18:33.740715 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.740778 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 11:18:33.741031 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
I20260811 11:18:33.741700 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.741565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51618)
0811 11:18:33.741638 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.741673 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260811 11:18:33.741887 11524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:40229: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab.
Entry for principal kudu/127.28.25.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab.
I20260811 11:18:33.754634 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.60:40229
--webserver_interface=127.28.25.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab
--principal=kudu/127.28.25.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:46705,127.28.25.61:45455,127.28.25.60:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:18:33.755581 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } attempt: 3
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
I20260811 11:18:33.764793 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.755992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:45455 (local address 127.0.0.1:60218)
0811 11:18:33.756060 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.756076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:33.756149 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:33.756301 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:33.756302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:33.756518 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:33.756606 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.756608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.757704 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.757705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.758533 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.758538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.759035 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.759036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.759138 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.759645 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:33.759651 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:33.761264 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:33.763590 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.763591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.763592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.763823 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.764067 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.764067 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.764068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.764128 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.764486 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:33.764487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:33.764540 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 11:18:33.764613 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 11:18:33.764713 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260811 11:18:33.764930 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.756068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.28.25.61:45455)
0811 11:18:33.756137 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.756139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.756148 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:18:33.756201 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.756202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.756213 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.756273 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.756717 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.757584 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.758644 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.758887 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.759001 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.761412 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.761413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.761414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.761423 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.763442 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.763912 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.763914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.763914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.763988 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.764200 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.764201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.764201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.764383 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.764466 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.764669 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.764775 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:18:33.765384 11438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:18:33.765799 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.765684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51624)
0811 11:18:33.765738 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.765776 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260811 11:18:33.766242 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.766155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51640)
0811 11:18:33.766202 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.766226 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 11:18:33.766371 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:40229: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
I20260811 11:18:33.791168 11524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } attempt: 1
I20260811 11:18:33.791813 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.791650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51642)
0811 11:18:33.791720 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.791774 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 11:18:33.792253 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.792150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51650)
0811 11:18:33.792202 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.792233 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 11:18:33.792433 11524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:40229: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
I20260811 11:18:33.797518 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } attempt: 1
I20260811 11:18:33.798152 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.798000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51658)
0811 11:18:33.798073 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.798118 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 11:18:33.798612 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.798510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51662)
0811 11:18:33.798562 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.798589 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:18:33.798799 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:40229: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
W20260811 11:18:33.850143 11531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:33.850409 11531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:33.850435 11531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:33.852315 11531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:18:33.852423 11531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:33.852447 11531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:33.852473 11531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:18:33.852504 11531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:18:33.854607 11531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:46705,127.28.25.61:45455,127.28.25.60:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab
--principal=kudu/127.28.25.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.60:40229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.25.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:33.855041 11531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:33.855324 11531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:18:33.860587 11524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } attempt: 2
I20260811 11:18:33.861208 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.860980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51672)
0811 11:18:33.861124 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.861159 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260811 11:18:33.861137 11538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:33.861236 11539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:33.861138 11541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:33.861428 11531 server_base.cc:1034] running on GCE node
I20260811 11:18:33.861590 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.861480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51686)
0811 11:18:33.861521 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.861543 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260811 11:18:33.861706 11524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:40229: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
I20260811 11:18:33.861770 11531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:33.862226 11531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:33.863408 11531 hybrid_clock.cc:648] HybridClock initialized: now 1786447113863385 us; error 44 us; skew 500 ppm
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:33.866847 11531 init.cc:377] Logged in from keytab as kudu/127.28.25.60@KRBTEST.COM (short username kudu)
I20260811 11:18:33.867551 11531 webserver.cc:460] Webserver started at http://127.28.25.60:43931/ using document root <none> and password file <none>
I20260811 11:18:33.867841 11531 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:33.867920 11531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:33.868090 11531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:33.869338 11531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/data/instance:
uuid: "b0f2d6e095b242ec8d97a14c4b1f5144"
format_stamp: "Formatted at 2026-08-11 11:18:33 on dist-test-slave-2w3w"
server_key: "557abf17d5be2284e0426f61c8d12a32"
server_key_iv: "e4f80c50724cf21bf2d6508957b6bca9"
server_key_version: "encryptionkey@0"
I20260811 11:18:33.869763 11531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/wal/instance:
uuid: "b0f2d6e095b242ec8d97a14c4b1f5144"
format_stamp: "Formatted at 2026-08-11 11:18:33 on dist-test-slave-2w3w"
server_key: "557abf17d5be2284e0426f61c8d12a32"
server_key_iv: "e4f80c50724cf21bf2d6508957b6bca9"
server_key_version: "encryptionkey@0"
I20260811 11:18:33.871505 11531 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 11:18:33.872488 11548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:33.872733 11531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:33.872880 11531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/wal
uuid: "b0f2d6e095b242ec8d97a14c4b1f5144"
format_stamp: "Formatted at 2026-08-11 11:18:33 on dist-test-slave-2w3w"
server_key: "557abf17d5be2284e0426f61c8d12a32"
server_key_iv: "e4f80c50724cf21bf2d6508957b6bca9"
server_key_version: "encryptionkey@0"
I20260811 11:18:33.873015 11531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:33.873986 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } attempt: 2
I20260811 11:18:33.874497 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.874351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51696)
0811 11:18:33.874416 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.874454 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 11:18:33.874853 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.874751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51710)
0811 11:18:33.874803 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.874829 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 11:18:33.874980 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:40229: Network error: Client connection negotiation failed: client connection to 127.28.25.60:40229: connect: Connection refused (error 111)
I20260811 11:18:33.883116 11531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:33.885640 11531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:33.885802 11531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:33.891309 11531 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.60:40229
I20260811 11:18:33.891368 11607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.60:40229 every 8 connection(s)
I20260811 11:18:33.891861 11531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
I20260811 11:18:33.892712 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 11531
I20260811 11:18:33.892827 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786446911996770-28772-0/minicluster-data/master-2/wal/instance
I20260811 11:18:33.892855 11608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:33.893054 28772 external_mini_cluster.cc:1428] Setting key 7f50953dff9408aeca68454be2fb0018
I20260811 11:18:33.895756 11608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:33.906075 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.896621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.28.25.62:46705)
0811 11:18:33.896694 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.896696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.896910 (+ 214us) server_negotiation.cc:407] Connection header received
0811 11:18:33.896957 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.896958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.896973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.897048 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.897814 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.898506 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.899677 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.899904 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.899977 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.902629 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.902631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.902632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.902642 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.904520 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.905002 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.905003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.905003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.905075 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.905299 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.905300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.905300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.905477 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.905570 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.905900 (+ 330us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.906003 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:18:33.906185 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.893667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.28.25.60:40229)
0811 11:18:33.893876 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.893884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.893914 (+ 30us) server_negotiation.cc:407] Connection header received
0811 11:18:33.894520 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.894528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.894580 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.894729 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.895791 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.897039 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.898098 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.898757 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.898840 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.901795 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.901804 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.901804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.901811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.904241 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.904685 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.904689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.904690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.904788 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.905034 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.905035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.905035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.905409 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.905516 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.905700 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.905918 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260811 11:18:33.906450 11612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.896546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46705 (local address 127.0.0.1:44816)
0811 11:18:33.896793 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.896843 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:33.896909 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:33.897169 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:33.897172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:33.897583 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:33.897675 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.897677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.898636 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.898637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.899559 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.899563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.900017 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.900018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.900113 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.900640 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:33.900658 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:33.902494 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:33.904619 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.904625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.904627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.904927 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.905141 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.905142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.905142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.905200 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.905569 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:33.905575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:33.905666 (+ 91us) client_negotiation.cc:769] Sending connection context
0811 11:18:33.905754 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 11:18:33.906365 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":477,"thread_start_us":149,"threads_started":1}
I20260811 11:18:33.907013 11608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:33.916211 11612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.907343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:45455 (local address 127.0.0.1:60222)
0811 11:18:33.907383 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.907392 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:33.907423 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:33.907602 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:33.907603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:33.907769 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:33.907821 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.907824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.908806 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.908807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.909664 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.909667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.910177 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.910177 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.910229 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.910833 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:33.910837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:33.912330 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:33.914806 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.914807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.914808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.915078 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.915367 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.915368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.915368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.915463 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.915784 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:33.915785 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:33.915921 (+ 136us) client_negotiation.cc:769] Sending connection context
0811 11:18:33.916010 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 11:18:33.916137 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260811 11:18:33.916275 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.907415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.28.25.61:45455)
0811 11:18:33.907469 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.907472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.907480 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:18:33.907508 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.907509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.907519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.907597 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.907923 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.908714 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.909768 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.910084 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.910165 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.912446 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.912448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.912448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.912458 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.914682 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.915167 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.915168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.915169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.915242 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.915547 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.915548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.915548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.915699 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.915774 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.916062 (+ 288us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.916159 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:18:33.916940 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.907255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.28.25.62:46705)
0811 11:18:33.907336 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.907338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.907350 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:18:33.907374 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.907375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.907386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.907458 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.908213 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.908957 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.909959 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.910278 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.910378 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.913230 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.913232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.913233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.913242 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.915220 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.915720 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.915722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.915722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.915785 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.916032 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.916034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.916034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.916180 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.916679 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.916728 (+ 49us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.916839 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 11:18:33.917013 11608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
I20260811 11:18:33.925896 11612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.917447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51716)
0811 11:18:33.917529 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.917541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:33.917597 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:33.917712 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:33.917713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:33.917889 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:33.917964 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.917966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.918655 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.918656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.919345 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.919348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.919735 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.919736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.919855 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.920370 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:33.920375 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:33.922088 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:33.924133 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.924134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.924135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.924406 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.924645 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.924646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.924646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.924736 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.925538 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:33.925539 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:33.925598 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 11:18:33.925679 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 11:18:33.925818 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260811 11:18:33.926201 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.917493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.28.25.60:40229)
0811 11:18:33.917535 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.917537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.917580 (+ 43us) server_negotiation.cc:407] Connection header received
0811 11:18:33.917651 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.917652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.917662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.917701 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.918034 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.918552 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.919446 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.919658 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.919715 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.922190 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.922192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.922193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.922202 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.923982 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.924500 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.924501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.924501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.924556 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.925230 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.925232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.925232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.925448 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.925550 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.926035 (+ 485us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.926130 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 11:18:33.927836 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.917755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.28.25.61:45455)
0811 11:18:33.917803 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.917805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.917928 (+ 123us) server_negotiation.cc:407] Connection header received
0811 11:18:33.917956 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.917957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.917967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.918027 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.919209 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.919845 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.920700 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.921005 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.921065 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.924163 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.924165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.924166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.924175 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.926466 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.926906 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.926907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.926907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.927018 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.927301 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.927306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.927306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.927443 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.927612 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.927690 (+ 78us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.927770 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:18:33.928241 11608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144: Bootstrap starting.
I20260811 11:18:33.929525 11608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:33.929709 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:18:33.929953 11608 log.cc:824] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:33.931047 11608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144: No bootstrap required, opened a new log
I20260811 11:18:33.932972 11608 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } }
I20260811 11:18:33.933120 11608 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:33.933142 11608 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f2d6e095b242ec8d97a14c4b1f5144, State: Initialized, Role: FOLLOWER
I20260811 11:18:33.933288 11608 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } }
I20260811 11:18:33.933637 11608 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:18:33.933606 11621 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } } }
I20260811 11:18:33.933718 11621 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 11:18:33.936515 11633 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:18:33.936587 11633 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:18:33.936071 11531 master_runner.cc:186] Error getting master registration for 127.28.25.62:46705: OK, instance_id { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" instance_seqno: 1786447113524553 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:18:33.939635 11531 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:18:33.940610 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.931743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.28.25.62:46705)
0811 11:18:33.931797 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.931799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.931810 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:18:33.931877 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.931878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.931895 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.931947 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.932616 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.933111 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.934217 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.934469 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.934556 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.936368 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.936369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.936370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.936379 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.938363 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.938820 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.938821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.938822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.939185 (+ 363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.939448 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.939449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.939449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.939600 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.940127 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.940286 (+ 159us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.940510 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 11:18:33.944203 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.931482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.28.25.61:45455)
0811 11:18:33.931531 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.931534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.935351 (+ 3817us) server_negotiation.cc:407] Connection header received
0811 11:18:33.935461 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.935463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.935475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.935509 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.936303 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.937014 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.938219 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.938515 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.938564 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.940589 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.940590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.940591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.940601 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.942354 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.942828 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.942829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.942830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.942899 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.943179 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.943180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.943180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.943308 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.944023 (+ 715us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.944062 (+ 39us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.944134 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 11:18:33.944279 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.931497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.28.25.60:40229)
0811 11:18:33.931577 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.931579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.935171 (+ 3592us) server_negotiation.cc:407] Connection header received
0811 11:18:33.935239 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.935240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.935254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.935314 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.937122 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.937901 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.938954 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.939336 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.939385 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.940849 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.940850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.940851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.940859 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.942457 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.942995 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.942997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.942997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.943058 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.943309 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.943310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.943310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.943504 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.943602 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.944075 (+ 473us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.944171 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 11:18:33.952869 11524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } attempt: 3
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
I20260811 11:18:33.961861 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.953349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51726)
0811 11:18:33.953425 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.953441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:33.953505 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:33.953640 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:33.953640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:33.953873 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:33.953935 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.953938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.954662 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.954663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.955568 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.955571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.956021 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.956022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.956084 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.956929 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:33.956935 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:33.958584 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:33.960518 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.960520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.960520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.960882 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.961129 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.961131 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.961131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.961201 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.961488 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:33.961489 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:33.961553 (+ 64us) client_negotiation.cc:769] Sending connection context
0811 11:18:33.961634 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 11:18:33.961759 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 11:18:33.961861 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.953437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.28.25.60:40229)
0811 11:18:33.953516 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.953519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.953531 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:18:33.953560 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.953561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.953570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.953632 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.954050 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.954558 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.955682 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.955933 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.956000 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.958701 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.958703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.958704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.958719 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.960420 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.960992 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.960993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.960994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.961070 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.961273 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.961274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.961274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.961413 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.961496 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.961661 (+ 165us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.961757 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:18:33.963877 11524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18: Bootstrap starting.
I20260811 11:18:33.964777 11524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:33.965139 11524 log.cc:824] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:33.966109 11524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18: No bootstrap required, opened a new log
I20260811 11:18:33.968842 11524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } }
I20260811 11:18:33.969024 11524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:33.969063 11524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6c48c1ea854cde8f233f0555147c18, State: Initialized, Role: FOLLOWER
I20260811 11:18:33.969197 11524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } }
I20260811 11:18:33.969521 11636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } } }
I20260811 11:18:33.969568 11524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:18:33.969653 11636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 11:18:33.971843 11648 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:18:33.971923 11648 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:18:33.974128 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } attempt: 3
I20260811 11:18:33.972195 11445 master_runner.cc:186] Error getting master registration for 127.28.25.62:46705: OK, instance_id { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" instance_seqno: 1786447113524553 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:18:33.974886 11445 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 11 11:18:33 dist-test-slave-2w3w krb5kdc[11348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
I20260811 11:18:33.982161 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.974464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:40229 (local address 127.0.0.1:51728)
0811 11:18:33.974571 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:33.974584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:33.974628 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:33.974993 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:33.974994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:33.975201 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:33.975250 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.975258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.976134 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.976135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.976856 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:33.976859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.977354 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.977355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:33.977412 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.977905 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:33.977909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:33.979353 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:33.981023 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.981024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.981024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.981215 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.981454 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:33.981455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:33.981455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:33.981522 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:33.981799 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:33.981800 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:33.981858 (+ 58us) client_negotiation.cc:769] Sending connection context
0811 11:18:33.981938 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 11:18:33.982055 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260811 11:18:33.982161 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:33.974725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.28.25.60:40229)
0811 11:18:33.974771 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:33.974774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:33.974783 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:18:33.974821 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:33.974822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:33.974843 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:33.974897 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:33.975373 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.976040 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.977018 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:33.977273 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:33.977354 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:33.979461 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:33.979463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:33.979463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:33.979472 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:33.980927 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.981306 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.981307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.981308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.981381 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:33.981588 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:33.981589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:33.981589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:33.981722 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:33.981796 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:33.981978 (+ 182us) server_negotiation.cc:299] Negotiation successful
0811 11:18:33.982060 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:18:33.984122 11438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c: Bootstrap starting.
I20260811 11:18:33.984862 11438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:33.985282 11438 log.cc:824] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:33.986483 11438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c: No bootstrap required, opened a new log
I20260811 11:18:33.988263 11438 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } }
I20260811 11:18:33.988415 11438 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:33.988464 11438 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea477beeee2a4f96b6239ed8e63ad92c, State: Initialized, Role: FOLLOWER
I20260811 11:18:33.988601 11438 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } }
I20260811 11:18:33.988941 11651 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } } }
I20260811 11:18:33.988998 11438 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:18:33.990492 11651 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [sys.catalog]: This master's current role is: FOLLOWER
W20260811 11:18:33.991065 11663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:18:33.991122 11663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:18:33.993906 11362 master_runner.cc:263] No leader master found from master ea477beeee2a4f96b6239ed8e63ad92c
I20260811 11:18:33.993966 11362 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260811 11:18:34.022101 11386 tablet.cc:2369] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:18:34.125471 11621 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:34.125625 11621 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } }
I20260811 11:18:34.125897 11621 leader_election.cc:288] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea477beeee2a4f96b6239ed8e63ad92c (127.28.25.62:46705), 7b6c48c1ea854cde8f233f0555147c18 (127.28.25.61:45455)
I20260811 11:18:34.126571 11409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" is_pre_election: true
I20260811 11:18:34.126731 11409 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f2d6e095b242ec8d97a14c4b1f5144 in term 0.
I20260811 11:18:34.127184 11492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6c48c1ea854cde8f233f0555147c18" is_pre_election: true
I20260811 11:18:34.127341 11492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f2d6e095b242ec8d97a14c4b1f5144 in term 0.
I20260811 11:18:34.127666 11549 leader_election.cc:302] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0f2d6e095b242ec8d97a14c4b1f5144, ea477beeee2a4f96b6239ed8e63ad92c; no voters:
I20260811 11:18:34.127885 11621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:18:34.127943 11621 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:18:34.127965 11621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:34.129175 11621 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } }
I20260811 11:18:34.129362 11621 leader_election.cc:288] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [CANDIDATE]: Term 1 election: Requested vote from peers ea477beeee2a4f96b6239ed8e63ad92c (127.28.25.62:46705), 7b6c48c1ea854cde8f233f0555147c18 (127.28.25.61:45455)
I20260811 11:18:34.129930 11492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6c48c1ea854cde8f233f0555147c18"
I20260811 11:18:34.130035 11492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:34.131115 11492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f2d6e095b242ec8d97a14c4b1f5144 in term 1.
I20260811 11:18:34.131399 11409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea477beeee2a4f96b6239ed8e63ad92c"
I20260811 11:18:34.131487 11409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:34.132325 11409 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f2d6e095b242ec8d97a14c4b1f5144 in term 1.
I20260811 11:18:34.132774 11552 leader_election.cc:302] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b6c48c1ea854cde8f233f0555147c18, b0f2d6e095b242ec8d97a14c4b1f5144; no voters:
I20260811 11:18:34.133222 11621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:18:34.133421 11621 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [term 1 LEADER]: Becoming Leader. State: Replica: b0f2d6e095b242ec8d97a14c4b1f5144, State: Running, Role: LEADER
I20260811 11:18:34.133524 11621 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } }
I20260811 11:18:34.133976 11621 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0f2d6e095b242ec8d97a14c4b1f5144. Latest consensus state: current_term: 1 leader_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } } }
I20260811 11:18:34.134059 11621 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:34.134680 11667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:18:34.135142 11667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260811 11:18:34.136468 11555 tablet.cc:2369] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:18:34.139086 11492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [term 1 FOLLOWER]: Refusing update from remote peer b0f2d6e095b242ec8d97a14c4b1f5144: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:18:34.139544 11621 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:18:34.141464 11636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0f2d6e095b242ec8d97a14c4b1f5144. Latest consensus state: current_term: 1 leader_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } } }
I20260811 11:18:34.141579 11636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:18:34.142174 11409 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [term 1 FOLLOWER]: Refusing update from remote peer b0f2d6e095b242ec8d97a14c4b1f5144: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:18:34.142522 11621 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:18:34.144223 11651 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0f2d6e095b242ec8d97a14c4b1f5144. Latest consensus state: current_term: 1 leader_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } } }
I20260811 11:18:34.144361 11651 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:18:34.144304 11665 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } } }
I20260811 11:18:34.144394 11665 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:34.144865 11636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } } }
I20260811 11:18:34.144975 11636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:18:34.145175 11668 mvcc.cc:205] Tried to move back new op lower bound from 7317287379510648832 to 7317287379491663872. Current Snapshot: MvccSnapshot[applied={T|T < 7317287379510648832}]
I20260811 11:18:34.145416 11621 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } } }
I20260811 11:18:34.145517 11621 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144 [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:34.145923 11669 mvcc.cc:205] Tried to move back new op lower bound from 7317287379510648832 to 7317287379491663872. Current Snapshot: MvccSnapshot[applied={T|T < 7317287379510648832}]
I20260811 11:18:34.145953 11651 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea477beeee2a4f96b6239ed8e63ad92c" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46705 } } peers { permanent_uuid: "7b6c48c1ea854cde8f233f0555147c18" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 45455 } } peers { permanent_uuid: "b0f2d6e095b242ec8d97a14c4b1f5144" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 40229 } } }
I20260811 11:18:34.146037 11651 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:18:34.146469 11667 catalog_manager.cc:1347] Generated new cluster ID: d7d7d3416a5343be92b69894c3c2a2b0
I20260811 11:18:34.146523 11667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:18:34.146790 11671 mvcc.cc:205] Tried to move back new op lower bound from 7317287379510648832 to 7317287379491663872. Current Snapshot: MvccSnapshot[applied={T|T < 7317287379510648832}]
I20260811 11:18:34.172866 11667 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:18:34.173499 11667 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:18:34.196098 11667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0f2d6e095b242ec8d97a14c4b1f5144: Generated new TSK 0
I20260811 11:18:34.196650 11667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:18:34.972451 11648 catalog_manager.cc:1259] Loaded cluster ID: d7d7d3416a5343be92b69894c3c2a2b0
I20260811 11:18:34.972545 11648 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18: loading cluster ID for follower catalog manager: success
I20260811 11:18:34.973424 11648 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18: acquiring CA information for follower catalog manager: success
I20260811 11:18:34.973845 11648 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b6c48c1ea854cde8f233f0555147c18: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:18:34.991629 11663 catalog_manager.cc:1259] Loaded cluster ID: d7d7d3416a5343be92b69894c3c2a2b0
I20260811 11:18:34.991717 11663 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c: loading cluster ID for follower catalog manager: success
I20260811 11:18:34.992697 11663 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c: acquiring CA information for follower catalog manager: success
I20260811 11:18:34.993111 11663 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea477beeee2a4f96b6239ed8e63ad92c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:18:35.251391 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 11362
I20260811 11:18:35.260855 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 11445
I20260811 11:18:35.270123 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 11531
2026-08-11T11:18:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3184 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:18:35 dist-test-slave-2w3w krb5kdc[11686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:18:35 dist-test-slave-2w3w krb5kdc[11686](info): set up 2 sockets
Aug 11 11:18:35 dist-test-slave-2w3w krb5kdc[11686](info): commencing operation
krb5kdc: starting...
W20260811 11:18:36.426889 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:18:36 dist-test-slave-2w3w krb5kdc[11686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:18:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:18:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:18:36.571334 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:34431
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:34431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:36.664196 11701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:36.664460 11701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:36.664489 11701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:36.665943 11701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:18:36.666008 11701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:36.666024 11701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:36.666038 11701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:18:36.666050 11701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:18:36.667852 11701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:34431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:34431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:36.668185 11701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:36.668462 11701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:36.674165 11706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:36.674104 11707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:36.674115 11709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:36.674540 11701 server_base.cc:1034] running on GCE node
I20260811 11:18:36.674829 11701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:36.675232 11701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:36.676429 11701 hybrid_clock.cc:648] HybridClock initialized: now 1786447116676405 us; error 48 us; skew 500 ppm
Aug 11 11:18:36 dist-test-slave-2w3w krb5kdc[11686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447116, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:36.679605 11701 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:18:36.680163 11701 webserver.cc:460] Webserver started at http://127.28.25.62:34787/ using document root <none> and password file <none>
I20260811 11:18:36.680433 11701 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:36.680483 11701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:36.680637 11701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:36.681880 11701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "ad1f15fdd1044c648fb2750e3391a23f"
format_stamp: "Formatted at 2026-08-11 11:18:36 on dist-test-slave-2w3w"
server_key: "d4e08b435d2e258a6ac44477e346083e"
server_key_iv: "75a5c5ac136a0b9c8dccfb3a8a7fb4a3"
server_key_version: "encryptionkey@0"
I20260811 11:18:36.682287 11701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "ad1f15fdd1044c648fb2750e3391a23f"
format_stamp: "Formatted at 2026-08-11 11:18:36 on dist-test-slave-2w3w"
server_key: "d4e08b435d2e258a6ac44477e346083e"
server_key_iv: "75a5c5ac136a0b9c8dccfb3a8a7fb4a3"
server_key_version: "encryptionkey@0"
I20260811 11:18:36.684100 11701 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 11:18:36.685194 11718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:36.685448 11701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 11:18:36.685570 11701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "ad1f15fdd1044c648fb2750e3391a23f"
format_stamp: "Formatted at 2026-08-11 11:18:36 on dist-test-slave-2w3w"
server_key: "d4e08b435d2e258a6ac44477e346083e"
server_key_iv: "75a5c5ac136a0b9c8dccfb3a8a7fb4a3"
server_key_version: "encryptionkey@0"
I20260811 11:18:36.685672 11701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:36.693936 11701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:36.697227 11701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:36.697469 11701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:36.703235 11701 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:34431
I20260811 11:18:36.703331 11778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:34431 every 8 connection(s)
I20260811 11:18:36.703758 11701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:18:36.705209 11779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:36.708350 11779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f: Bootstrap starting.
I20260811 11:18:36.709156 11779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:36.709170 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 11701
I20260811 11:18:36.709280 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:18:36.709554 28772 external_mini_cluster.cc:1428] Setting key fecaa16977040fa040ee6e5dc96c2214
I20260811 11:18:36.709575 11779 log.cc:824] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:36.710528 11779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f: No bootstrap required, opened a new log
I20260811 11:18:36.712749 11779 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1f15fdd1044c648fb2750e3391a23f" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 34431 } }
I20260811 11:18:36.712978 11779 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:36.713023 11779 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1f15fdd1044c648fb2750e3391a23f, State: Initialized, Role: FOLLOWER
I20260811 11:18:36.713167 11779 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1f15fdd1044c648fb2750e3391a23f" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 34431 } }
I20260811 11:18:36.713267 11779 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:18:36.713325 11779 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:18:36.713380 11779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:36.714241 11779 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1f15fdd1044c648fb2750e3391a23f" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 34431 } }
I20260811 11:18:36.714362 11779 leader_election.cc:302] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad1f15fdd1044c648fb2750e3391a23f; no voters:
I20260811 11:18:36.714546 11779 leader_election.cc:288] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:18:36.714672 11786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:18:36.714879 11779 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:18:36.714906 11786 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [term 1 LEADER]: Becoming Leader. State: Replica: ad1f15fdd1044c648fb2750e3391a23f, State: Running, Role: LEADER
I20260811 11:18:36.715161 11786 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1f15fdd1044c648fb2750e3391a23f" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 34431 } }
I20260811 11:18:36.715579 11787 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad1f15fdd1044c648fb2750e3391a23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1f15fdd1044c648fb2750e3391a23f" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 34431 } } }
I20260811 11:18:36.715587 11789 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad1f15fdd1044c648fb2750e3391a23f. Latest consensus state: current_term: 1 leader_uuid: "ad1f15fdd1044c648fb2750e3391a23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1f15fdd1044c648fb2750e3391a23f" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 34431 } } }
I20260811 11:18:36.715755 11787 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:36.715762 11789 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:36.716609 11796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:18:36.717352 11796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 11 11:18:36 dist-test-slave-2w3w krb5kdc[11686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:36.720990 11796 catalog_manager.cc:1347] Generated new cluster ID: ae6d584d4e1b4488827965acf5b2077a
I20260811 11:18:36.721125 11796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:18:36.724288 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:36.710147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.28.25.62:34431)
0811 11:18:36.710353 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:36.710364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:36.710985 (+ 621us) server_negotiation.cc:407] Connection header received
0811 11:18:36.711924 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:36.711935 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:36.712020 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:36.712195 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:36.712873 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:36.714684 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:36.715992 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:36.716651 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:36.716929 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:36.720075 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:36.720080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:36.720080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:36.720096 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:36.722105 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:36.722559 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:36.722562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:36.722563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:36.722672 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:36.722907 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:36.722908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:36.722908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:36.723246 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:36.723875 (+ 629us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:36.723981 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 11:18:36.724111 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260811 11:18:36.725306 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:18:36.741218 11796 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:18:36.741954 11796 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:18:36.744967 11796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad1f15fdd1044c648fb2750e3391a23f: Generated new TSK 0
I20260811 11:18:36.745206 11796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:18:36.780018 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 11701
2026-08-11T11:18:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1507 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:18:36 dist-test-slave-2w3w krb5kdc[11809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:18:36 dist-test-slave-2w3w krb5kdc[11809](info): set up 2 sockets
Aug 11 11:18:36 dist-test-slave-2w3w krb5kdc[11809](info): commencing operation
krb5kdc: starting...
W20260811 11:18:37.913641 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:18:38 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:18:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:18:38.058455 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39205
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:39205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:38.145287 11822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:38.145537 11822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:38.145570 11822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:38.147073 11822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:18:38.147136 11822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:38.147151 11822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:38.147166 11822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:18:38.147177 11822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:18:38.148932 11822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:39205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:38.149287 11822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:38.149565 11822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:38.155025 11832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:38.155004 11835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:38.155050 11822 server_base.cc:1034] running on GCE node
W20260811 11:18:38.155014 11831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:38.155604 11822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:38.155930 11822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:38.157114 11822 hybrid_clock.cc:648] HybridClock initialized: now 1786447118157095 us; error 29 us; skew 500 ppm
Aug 11 11:18:38 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447118, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:38.163601 11822 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:18:38.164129 11822 webserver.cc:460] Webserver started at http://127.28.25.62:35295/ using document root <none> and password file <none>
I20260811 11:18:38.164381 11822 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:38.164455 11822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:38.164615 11822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:38.165766 11822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "adb8ab01dc854a73894c9921aa6f10a0"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "7a25e13f0555cc0b9c10f6d24cf3b925"
server_key_iv: "d95c5a7b68500465d3f7a000ce79c514"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.166177 11822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "adb8ab01dc854a73894c9921aa6f10a0"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "7a25e13f0555cc0b9c10f6d24cf3b925"
server_key_iv: "d95c5a7b68500465d3f7a000ce79c514"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.167753 11822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:18:38.168679 11845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:38.168985 11822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:38.169099 11822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "adb8ab01dc854a73894c9921aa6f10a0"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "7a25e13f0555cc0b9c10f6d24cf3b925"
server_key_iv: "d95c5a7b68500465d3f7a000ce79c514"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.169183 11822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:38.182070 11822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:38.184705 11822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:38.184916 11822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:38.189220 11822 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:39205
I20260811 11:18:38.189307 11901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:39205 every 8 connection(s)
I20260811 11:18:38.189703 11822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:18:38.190901 11903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:38.193434 11903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Bootstrap starting.
I20260811 11:18:38.194062 11903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:38.194327 11903 log.cc:824] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:38.195246 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 11822
I20260811 11:18:38.195286 11903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: No bootstrap required, opened a new log
I20260811 11:18:38.195399 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:18:38.195695 28772 external_mini_cluster.cc:1428] Setting key 500fcb152f7fe621b63adcf866d9930f
I20260811 11:18:38.198863 11903 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:38.199152 11903 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:38.199229 11903 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb8ab01dc854a73894c9921aa6f10a0, State: Initialized, Role: FOLLOWER
I20260811 11:18:38.199470 11903 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:38.199604 11908 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:38.199712 11908 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:38.199582 11903 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:18:38.199817 11908 leader_election.cc:302] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb8ab01dc854a73894c9921aa6f10a0; no voters:
I20260811 11:18:38.199862 11903 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:18:38.199936 11903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:38.200012 11908 leader_election.cc:288] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260811 11:18:38.200802 11903 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:38.200910 11903 leader_election.cc:302] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb8ab01dc854a73894c9921aa6f10a0; no voters:
I20260811 11:18:38.200919 11909 raft_consensus.cc:2741] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260811 11:18:38.201001 11903 leader_election.cc:288] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:18:38.201185 11908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:18:38.201303 11908 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 1 LEADER]: Becoming Leader. State: Replica: adb8ab01dc854a73894c9921aa6f10a0, State: Running, Role: LEADER
I20260811 11:18:38.201355 11903 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:18:38.201434 11908 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:38.201804 11909 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adb8ab01dc854a73894c9921aa6f10a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } } }
I20260811 11:18:38.201900 11909 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:38.202469 11910 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb8ab01dc854a73894c9921aa6f10a0. Latest consensus state: current_term: 1 leader_uuid: "adb8ab01dc854a73894c9921aa6f10a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } } }
I20260811 11:18:38.202629 11910 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: This master's current role is: LEADER
W20260811 11:18:38.203315 11924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:18:38.203380 11924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:18:38.203429 11915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:18:38.203634 11915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 11 11:18:38 dist-test-slave-2w3w krb5kdc[11809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:38.205613 11915 catalog_manager.cc:1347] Generated new cluster ID: 8b94a3019e214f569f948f4b5f88a6e9
I20260811 11:18:38.205684 11915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:18:38.210004 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.196376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.28.25.62:39205)
0811 11:18:38.196696 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.196706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.196741 (+ 35us) server_negotiation.cc:407] Connection header received
0811 11:18:38.197763 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.197775 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.197877 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.198046 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:38.199192 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.200637 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.201875 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.202511 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.203021 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.205974 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:38.205978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:38.205979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:38.205992 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:38.208150 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:38.208592 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:38.208596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:38.208596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:38.208715 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:38.208944 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:38.208945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:38.208945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:38.209282 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:38.209462 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.209579 (+ 117us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.209736 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":147,"threads_started":1}
I20260811 11:18:38.213351 11915 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:18:38.213891 11915 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:18:38.215862 11915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Generated new TSK 0
I20260811 11:18:38.216104 11915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:18:38.252401 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39205
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:38.342276 11930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:38.342512 11930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:38.342541 11930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:38.343848 11930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:38.343906 11930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:38.343935 11930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:18:38.345469 11930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:38.345685 11930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:38.345898 11930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:38.351619 11935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:38.351600 11936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:38.351649 11930 server_base.cc:1034] running on GCE node
W20260811 11:18:38.351600 11938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:38.352142 11930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:38.352401 11930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:38.353595 11930 hybrid_clock.cc:648] HybridClock initialized: now 1786447118353575 us; error 42 us; skew 500 ppm
Aug 11 11:18:38 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447118, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:38.356624 11930 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:18:38.357088 11930 webserver.cc:460] Webserver started at http://127.28.25.1:36543/ using document root <none> and password file <none>
I20260811 11:18:38.357287 11930 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:38.357342 11930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:38.357441 11930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:38.358585 11930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "e93aad5aa4844bc08cb3d5d64a4b262f"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "ec44e00ff57330148b8c8b9331f3804d"
server_key_iv: "15276159fde0a2117433fcadfc65538e"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.358924 11930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "e93aad5aa4844bc08cb3d5d64a4b262f"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "ec44e00ff57330148b8c8b9331f3804d"
server_key_iv: "15276159fde0a2117433fcadfc65538e"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.360692 11930 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260811 11:18:38.361613 11945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:38.361917 11930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:38.362007 11930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "e93aad5aa4844bc08cb3d5d64a4b262f"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "ec44e00ff57330148b8c8b9331f3804d"
server_key_iv: "15276159fde0a2117433fcadfc65538e"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.362121 11930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:38.378031 11930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:38.380692 11930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:38.380856 11930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:38.381160 11930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:38.381520 11930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:38.381577 11930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:38.381613 11930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:38.381641 11930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:38.388289 11930 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:36293
I20260811 11:18:38.388393 12077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:36293 every 8 connection(s)
I20260811 11:18:38.388765 11930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:18:38.388970 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 11930
I20260811 11:18:38.389101 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:18:38.389372 28772 external_mini_cluster.cc:1428] Setting key c66eca25df591a3ea1a6a1b91bd9aa67
Aug 11 11:18:38 dist-test-slave-2w3w krb5kdc[11809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447118, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
I20260811 11:18:38.401958 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.389810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:37933 (local address 127.28.25.62:39205)
0811 11:18:38.389909 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.389911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.390088 (+ 177us) server_negotiation.cc:407] Connection header received
0811 11:18:38.390940 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.390941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.391097 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.391165 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:38.392652 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.393505 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.394676 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.395118 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.395200 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.398116 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:38.398118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:38.398118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:38.398129 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:38.399990 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:38.400582 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:38.400583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:38.400584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:38.400644 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:38.401002 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:38.401004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:38.401004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:38.401153 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:38.401216 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.401705 (+ 489us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.401822 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 11:18:38.402254 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.389643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:37933)
0811 11:18:38.389958 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:38.390008 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:38.390779 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:38.391275 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:38.391281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:38.392206 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:38.392529 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.392535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.393634 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.393635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.394561 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.394563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.395244 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.395245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.395387 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.396056 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:38.396067 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:38.397982 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:38.400142 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:38.400154 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:38.400158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:38.400474 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:38.400765 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:38.400766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:38.400766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:38.400911 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:38.401240 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:38.401248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:38.401526 (+ 278us) client_negotiation.cc:769] Sending connection context
0811 11:18:38.401664 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 11:18:38.401954 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":104,"threads_started":1}
I20260811 11:18:38.403056 12078 heartbeater.cc:347] Connected to a master server at 127.28.25.62:39205
I20260811 11:18:38.403182 12078 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:38.403425 12078 heartbeater.cc:511] Master 127.28.25.62:39205 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
I20260811 11:18:38.404282 11864 ts_manager.cc:194] Registered new tserver with Master: e93aad5aa4844bc08cb3d5d64a4b262f (127.28.25.1:36293)
I20260811 11:18:38.405292 11864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:37933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:18:38.422709 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39205
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:38.512091 12084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:38.512415 12084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:38.512467 12084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:38.513705 12084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:38.513777 12084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:38.513825 12084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:18:38.515478 12084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:38.515781 12084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:38.516036 12084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:38.521471 12092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:38.521493 12090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:38.521490 12089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:38.521907 12084 server_base.cc:1034] running on GCE node
I20260811 11:18:38.522109 12084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:38.522396 12084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:38.523597 12084 hybrid_clock.cc:648] HybridClock initialized: now 1786447118523576 us; error 31 us; skew 500 ppm
Aug 11 11:18:38 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447118, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:38.526423 12084 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:18:38.527020 12084 webserver.cc:460] Webserver started at http://127.28.25.2:34297/ using document root <none> and password file <none>
I20260811 11:18:38.527273 12084 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:38.527344 12084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:38.527493 12084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:38.528607 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "e9ffae368ba74ac8b9245c80111586e3"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "f0c0e85aae0d4a681a26a7307ac6f880"
server_key_iv: "ce067474f6650d10cf9467d353443a06"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.529001 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "e9ffae368ba74ac8b9245c80111586e3"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "f0c0e85aae0d4a681a26a7307ac6f880"
server_key_iv: "ce067474f6650d10cf9467d353443a06"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.530534 12084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:18:38.531466 12099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:38.531661 12084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:18:38.531781 12084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "e9ffae368ba74ac8b9245c80111586e3"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "f0c0e85aae0d4a681a26a7307ac6f880"
server_key_iv: "ce067474f6650d10cf9467d353443a06"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.531859 12084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:38.545851 12084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:38.548619 12084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:38.548875 12084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:38.549144 12084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:38.549541 12084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:38.549598 12084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:38.549623 12084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:38.549635 12084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:38.556344 12084 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:43589
I20260811 11:18:38.556421 12229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:43589 every 8 connection(s)
I20260811 11:18:38.556806 12084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:18:38.559803 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 12084
I20260811 11:18:38.559948 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:18:38.560164 28772 external_mini_cluster.cc:1428] Setting key daeac27084276042300c8d1a50ecd2aa
Aug 11 11:18:38 dist-test-slave-2w3w krb5kdc[11809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447118, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:38.569279 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.557774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:37157 (local address 127.28.25.62:39205)
0811 11:18:38.557905 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.557908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.558101 (+ 193us) server_negotiation.cc:407] Connection header received
0811 11:18:38.558820 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.558821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.558837 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.558894 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:38.560111 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.560922 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.562664 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.562937 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.563099 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.565995 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:38.565997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:38.565998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:38.566006 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:38.567669 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:38.568161 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:38.568163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:38.568163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:38.568227 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:38.568522 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:38.568523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:38.568524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:38.568656 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:38.568740 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.569065 (+ 325us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.569165 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260811 11:18:38.569511 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.557688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:37157)
0811 11:18:38.557961 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:38.558011 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:38.558570 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:38.559000 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:38.559004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:38.559730 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:38.559997 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.560005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.561082 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.561083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.562526 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.562530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.563115 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.563117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.563302 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.564000 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:38.564009 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:38.565841 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:38.567774 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:38.567786 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:38.567789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:38.568068 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:38.568301 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:38.568302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:38.568302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:38.568446 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:38.568728 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:38.568737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:38.568910 (+ 173us) client_negotiation.cc:769] Sending connection context
0811 11:18:38.569029 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 11:18:38.569223 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":113,"threads_started":1}
I20260811 11:18:38.570048 12230 heartbeater.cc:347] Connected to a master server at 127.28.25.62:39205
I20260811 11:18:38.570154 12230 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:38.570382 12230 heartbeater.cc:511] Master 127.28.25.62:39205 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
I20260811 11:18:38.570820 11864 ts_manager.cc:194] Registered new tserver with Master: e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589)
I20260811 11:18:38.571259 11864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:37157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:18:38.596172 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39205
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:38.687852 12235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:38.688138 12235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:38.688169 12235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:38.689590 12235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:38.689649 12235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:38.689682 12235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:18:38.691375 12235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:38.691619 12235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:38.691847 12235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:38.698091 12240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:38.698045 12241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:38.698099 12235 server_base.cc:1034] running on GCE node
W20260811 11:18:38.698055 12243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:38.698616 12235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:38.698942 12235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:38.700174 12235 hybrid_clock.cc:648] HybridClock initialized: now 1786447118700125 us; error 71 us; skew 500 ppm
Aug 11 11:18:38 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447118, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:38.708828 12235 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:18:38.709380 12235 webserver.cc:460] Webserver started at http://127.28.25.3:44765/ using document root <none> and password file <none>
I20260811 11:18:38.709633 12235 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:38.709705 12235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:38.709867 12235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:38.711102 12235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "218c9e30682044c3bab931618da129cb"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "b9764fa3f7311338083796b2bde883dc"
server_key_iv: "8941f156f64e94ef96a15ffe1555c074"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.711506 12235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "218c9e30682044c3bab931618da129cb"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "b9764fa3f7311338083796b2bde883dc"
server_key_iv: "8941f156f64e94ef96a15ffe1555c074"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.713235 12235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:18:38.714185 12251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:38.714543 12235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:38.714668 12235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "218c9e30682044c3bab931618da129cb"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "b9764fa3f7311338083796b2bde883dc"
server_key_iv: "8941f156f64e94ef96a15ffe1555c074"
server_key_version: "encryptionkey@0"
I20260811 11:18:38.714850 12235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:38.756876 12235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:38.759500 12235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:38.759685 12235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:38.759953 12235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:38.760301 12235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:38.760357 12235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:38.760401 12235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:38.760447 12235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:38.767650 12235 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:44201
I20260811 11:18:38.767707 12382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:44201 every 8 connection(s)
I20260811 11:18:38.768074 12235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:18:38.774134 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 12235
I20260811 11:18:38.774269 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:18:38.774454 28772 external_mini_cluster.cc:1428] Setting key 935c6589dd1b3912221dbc9897c2a9f6
Aug 11 11:18:38 dist-test-slave-2w3w krb5kdc[11809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447118, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:38.779587 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.768889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:41787 (local address 127.28.25.62:39205)
0811 11:18:38.768961 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.768964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.769215 (+ 251us) server_negotiation.cc:407] Connection header received
0811 11:18:38.769746 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.769748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.769760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.769848 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:38.771057 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.771595 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.772945 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.773142 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.773229 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.776440 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:38.776441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:38.776441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:38.776450 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:38.778038 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:38.778512 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:38.778513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:38.778514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:38.778577 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:38.778838 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:38.778839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:38.778839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:38.779030 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:38.779100 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.779438 (+ 338us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.779518 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 11:18:38.779835 12385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.768875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:41787)
0811 11:18:38.769060 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:38.769104 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:38.769629 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:38.770017 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:38.770021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:38.770641 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:38.770869 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.770873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.771755 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.771756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.772842 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.772846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.773251 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.773254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.773386 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.773986 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:38.773998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:38.776338 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:38.778128 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:38.778137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:38.778139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:38.778413 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:38.778652 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:38.778653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:38.778653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:38.778768 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:38.779111 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:38.779117 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:38.779266 (+ 149us) client_negotiation.cc:769] Sending connection context
0811 11:18:38.779405 (+ 139us) client_negotiation.cc:240] Negotiation successful
0811 11:18:38.779551 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260811 11:18:38.780509 12383 heartbeater.cc:347] Connected to a master server at 127.28.25.62:39205
I20260811 11:18:38.780635 12383 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:38.780851 12383 heartbeater.cc:511] Master 127.28.25.62:39205 requested a full tablet report, sending...
I20260811 11:18:38.781276 11864 ts_manager.cc:194] Registered new tserver with Master: 218c9e30682044c3bab931618da129cb (127.28.25.3:44201)
I20260811 11:18:38.781635 11864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:41787
I20260811 11:18:38.786556 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:18:38.794899 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.788130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.28.25.62:39205)
0811 11:18:38.788216 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.788219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.788308 (+ 89us) server_negotiation.cc:407] Connection header received
0811 11:18:38.788400 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.788401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.788413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.788466 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:38.789134 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.789651 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.790408 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.790625 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.790706 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.792035 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:38.792037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:38.792038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:38.792047 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:38.793547 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:38.793949 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:38.793950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:38.793951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:38.794020 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:38.794234 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:38.794234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:38.794235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:38.794416 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:38.794535 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.794682 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.794783 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:18:38.796593 11864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:18:38.798185 11864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:18:38.805661 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.800979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.28.25.3:44201)
0811 11:18:38.801383 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.801388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.801399 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:18:38.801442 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.801445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.801506 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.801810 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:38.801906 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.803123 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.804490 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.805188 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.805254 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.805321 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.805521 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.805567 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":106,"threads_started":1}
I20260811 11:18:38.805691 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.801060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36293 (local address 127.0.0.1:34088)
0811 11:18:38.801276 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:38.801285 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:38.801328 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:38.801736 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:38.801736 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:38.801738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:38.801785 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.801786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.803267 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.803269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.804267 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.804271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.805275 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.805277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.805386 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.805404 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 11:18:38.805492 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 11:18:38.805565 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260811 11:18:38.805691 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.800850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:44201 (local address 127.0.0.1:48478)
0811 11:18:38.801184 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:38.801197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:38.801251 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:38.801634 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:38.801637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:38.801640 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:38.801724 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.801726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.803229 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.803230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.804399 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.804400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.805315 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.805316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.805389 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.805404 (+ 15us) client_negotiation.cc:769] Sending connection context
0811 11:18:38.805492 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 11:18:38.805565 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":41,"thread_start_us":111,"threads_started":1}
I20260811 11:18:38.806348 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.801303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.28.25.1:36293)
0811 11:18:38.801486 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.801491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.801501 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:18:38.801538 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.801540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.801554 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.801639 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:38.801856 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.803123 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.804374 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.805167 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.805255 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.805321 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.806135 (+ 814us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.806191 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
I20260811 11:18:38.806562 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.800810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:43589 (local address 127.0.0.1:38402)
0811 11:18:38.801058 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:38.801105 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:38.801177 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:38.801667 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:38.801668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:38.801671 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:38.801733 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.801735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.802889 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.802891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.804392 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.804395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.806316 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.806317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.806386 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.806390 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:18:38.806446 (+ 56us) client_negotiation.cc:240] Negotiation successful
0811 11:18:38.806490 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260811 11:18:38.806838 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.800983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.28.25.2:43589)
0811 11:18:38.801178 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.801183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.801194 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:18:38.801534 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.801537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.801554 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.801812 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:38.801890 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.802786 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.805368 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.806204 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.806311 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.806377 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.806706 (+ 329us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.806757 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260811 11:18:38.807140 12308 tablet_service.cc:1467] Processing CreateTablet for tablet 2dccede190e2437b8bcef1332b5e169d (DEFAULT_TABLE table=test-table [id=28a2cc87e9ef4e2ca51728d85ea7bb7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:38.807128 12006 tablet_service.cc:1467] Processing CreateTablet for tablet 2dccede190e2437b8bcef1332b5e169d (DEFAULT_TABLE table=test-table [id=28a2cc87e9ef4e2ca51728d85ea7bb7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:38.807508 12308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dccede190e2437b8bcef1332b5e169d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:38.807533 12006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dccede190e2437b8bcef1332b5e169d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:38.807592 12155 tablet_service.cc:1467] Processing CreateTablet for tablet 2dccede190e2437b8bcef1332b5e169d (DEFAULT_TABLE table=test-table [id=28a2cc87e9ef4e2ca51728d85ea7bb7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:38.807904 12155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dccede190e2437b8bcef1332b5e169d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:38.811548 12400 tablet_bootstrap.cc:492] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: Bootstrap starting.
I20260811 11:18:38.813723 12400 tablet_bootstrap.cc:654] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:38.814736 12400 log.cc:824] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:38.814929 12401 tablet_bootstrap.cc:492] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: Bootstrap starting.
I20260811 11:18:38.815591 12402 tablet_bootstrap.cc:492] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Bootstrap starting.
I20260811 11:18:38.815902 12400 tablet_bootstrap.cc:492] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: No bootstrap required, opened a new log
I20260811 11:18:38.815929 12401 tablet_bootstrap.cc:654] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:38.815994 12400 ts_tablet_manager.cc:1397] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260811 11:18:38.816387 12402 tablet_bootstrap.cc:654] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:38.816591 12401 log.cc:824] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:38.817411 12402 log.cc:824] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:38.818471 12401 tablet_bootstrap.cc:492] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: No bootstrap required, opened a new log
I20260811 11:18:38.818612 12401 ts_tablet_manager.cc:1397] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260811 11:18:38.818661 12400 raft_consensus.cc:348] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:38.818864 12400 raft_consensus.cc:374] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:38.818907 12400 raft_consensus.cc:729] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218c9e30682044c3bab931618da129cb, State: Initialized, Role: FOLLOWER
I20260811 11:18:38.819128 12400 consensus_queue.cc:260] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:38.819470 12400 ts_tablet_manager.cc:1428] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260811 11:18:38.819609 12383 heartbeater.cc:503] Master 127.28.25.62:39205 was elected leader, sending a full tablet report...
I20260811 11:18:38.821039 12402 tablet_bootstrap.cc:492] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: No bootstrap required, opened a new log
I20260811 11:18:38.821194 12402 ts_tablet_manager.cc:1397] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260811 11:18:38.821014 12401 raft_consensus.cc:348] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:38.821360 12401 raft_consensus.cc:374] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:38.821424 12401 raft_consensus.cc:729] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ffae368ba74ac8b9245c80111586e3, State: Initialized, Role: FOLLOWER
I20260811 11:18:38.821610 12401 consensus_queue.cc:260] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:38.822010 12401 ts_tablet_manager.cc:1428] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:18:38.822027 12230 heartbeater.cc:503] Master 127.28.25.62:39205 was elected leader, sending a full tablet report...
I20260811 11:18:38.824656 12402 raft_consensus.cc:348] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:38.824908 12402 raft_consensus.cc:374] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:38.824959 12402 raft_consensus.cc:729] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93aad5aa4844bc08cb3d5d64a4b262f, State: Initialized, Role: FOLLOWER
I20260811 11:18:38.825098 12402 consensus_queue.cc:260] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:38.825471 12402 ts_tablet_manager.cc:1428] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260811 11:18:38.825502 12078 heartbeater.cc:503] Master 127.28.25.62:39205 was elected leader, sending a full tablet report...
W20260811 11:18:38.889684 12079 tablet.cc:2369] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:18:38.960769 12408 raft_consensus.cc:482] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:38.960932 12408 raft_consensus.cc:504] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:38.961467 12408 leader_election.cc:288] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218c9e30682044c3bab931618da129cb (127.28.25.3:44201), e93aad5aa4844bc08cb3d5d64a4b262f (127.28.25.1:36293)
I20260811 11:18:38.964951 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.961753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:39435 (local address 127.28.25.1:36293)
0811 11:18:38.961811 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.961813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.962022 (+ 209us) server_negotiation.cc:407] Connection header received
0811 11:18:38.962098 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.962099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.962113 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.962166 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:38.962375 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.962870 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.964087 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.964562 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.964647 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.964659 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.964823 (+ 164us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.964867 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 11:18:38.964965 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.961694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36293 (local address 127.28.25.2:39435)
0811 11:18:38.961927 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:38.961946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:38.962035 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:38.962169 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:38.962169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:38.962171 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:38.962290 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.962292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.962997 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.962998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.963974 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.963977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.964650 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.964651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.964739 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.964743 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:18:38.964815 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 11:18:38.964866 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":39,"thread_start_us":171,"threads_started":1}
I20260811 11:18:38.965034 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.961780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:44529 (local address 127.28.25.3:44201)
0811 11:18:38.961868 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.961871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.961880 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:18:38.961918 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.961919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.961930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.962006 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:38.962179 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.962852 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.964096 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.964615 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.964696 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.964709 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.964916 (+ 207us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.964962 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 11:18:38.965050 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.961669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:44201 (local address 127.28.25.2:44529)
0811 11:18:38.961738 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:38.961751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:38.961802 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:38.962013 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:38.962015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:38.962017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:38.962100 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.962102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.963111 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.963113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.963971 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:38.963975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.964709 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.964710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:38.964807 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.964810 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:18:38.964883 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 11:18:38.964934 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":6}
I20260811 11:18:38.965626 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dccede190e2437b8bcef1332b5e169d" candidate_uuid: "e9ffae368ba74ac8b9245c80111586e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" is_pre_election: true
I20260811 11:18:38.965637 12333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dccede190e2437b8bcef1332b5e169d" candidate_uuid: "e9ffae368ba74ac8b9245c80111586e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218c9e30682044c3bab931618da129cb" is_pre_election: true
I20260811 11:18:38.965822 12333 raft_consensus.cc:2454] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9ffae368ba74ac8b9245c80111586e3 in term 0.
I20260811 11:18:38.965818 12027 raft_consensus.cc:2454] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9ffae368ba74ac8b9245c80111586e3 in term 0.
I20260811 11:18:38.966188 12104 leader_election.cc:302] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e93aad5aa4844bc08cb3d5d64a4b262f, e9ffae368ba74ac8b9245c80111586e3; no voters:
I20260811 11:18:38.966452 12408 raft_consensus.cc:2781] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:18:38.966512 12408 raft_consensus.cc:482] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:18:38.966528 12408 raft_consensus.cc:3037] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:38.967340 12408 raft_consensus.cc:504] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:38.967527 12408 leader_election.cc:288] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [CANDIDATE]: Term 1 election: Requested vote from peers 218c9e30682044c3bab931618da129cb (127.28.25.3:44201), e93aad5aa4844bc08cb3d5d64a4b262f (127.28.25.1:36293)
I20260811 11:18:38.967726 12333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dccede190e2437b8bcef1332b5e169d" candidate_uuid: "e9ffae368ba74ac8b9245c80111586e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218c9e30682044c3bab931618da129cb"
I20260811 11:18:38.967821 12333 raft_consensus.cc:3037] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:38.968655 12333 raft_consensus.cc:2454] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9ffae368ba74ac8b9245c80111586e3 in term 1.
I20260811 11:18:38.968922 12101 leader_election.cc:302] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218c9e30682044c3bab931618da129cb, e9ffae368ba74ac8b9245c80111586e3; no voters:
I20260811 11:18:38.968797 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dccede190e2437b8bcef1332b5e169d" candidate_uuid: "e9ffae368ba74ac8b9245c80111586e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f"
I20260811 11:18:38.969058 12408 raft_consensus.cc:2781] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:18:38.969045 12027 raft_consensus.cc:3037] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:38.969282 12408 raft_consensus.cc:686] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 1 LEADER]: Becoming Leader. State: Replica: e9ffae368ba74ac8b9245c80111586e3, State: Running, Role: LEADER
I20260811 11:18:38.969406 12408 consensus_queue.cc:237] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:38.970167 12027 raft_consensus.cc:2454] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9ffae368ba74ac8b9245c80111586e3 in term 1.
I20260811 11:18:38.970412 11863 catalog_manager.cc:5495] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2). New cstate: current_term: 1 leader_uuid: "e9ffae368ba74ac8b9245c80111586e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } health_report { overall_health: UNKNOWN } } }
I20260811 11:18:38.989233 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:38.986446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38406 (local address 127.28.25.2:43589)
0811 11:18:38.986515 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:38.986517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:38.986524 (+ 7us) server_negotiation.cc:407] Connection header received
0811 11:18:38.986564 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:38.986565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:38.986588 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:38.986673 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:18:38.986850 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.987435 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.988307 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:38.988512 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:38.988589 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:38.988798 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:18:38.988955 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:18:38.989062 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:38.989083 (+ 21us) server_negotiation.cc:299] Negotiation successful
0811 11:18:38.989107 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 11:18:38.991683 12333 raft_consensus.cc:1261] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb [term 1 FOLLOWER]: Refusing update from remote peer e9ffae368ba74ac8b9245c80111586e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:18:38.992059 12027 raft_consensus.cc:1261] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Refusing update from remote peer e9ffae368ba74ac8b9245c80111586e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:18:38.992113 12408 consensus_queue.cc:1035] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:18:38.992458 12408 consensus_queue.cc:1035] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:18:38.995855 12415 mvcc.cc:205] Tried to move back new op lower bound from 7317287399386652672 to 7317287399299342336. Current Snapshot: MvccSnapshot[applied={T|T < 7317287399386652672 or (T in {7317287399386652672})}]
I20260811 11:18:38.997598 11863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:18:38.997720 11863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:18:38.998556 11863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Sending DeleteTablet for 3 replicas of tablet 2dccede190e2437b8bcef1332b5e169d
I20260811 11:18:38.999110 12155 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dccede190e2437b8bcef1332b5e169d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:38 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:38402
I20260811 11:18:38.999166 12006 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dccede190e2437b8bcef1332b5e169d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:38 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:34088
I20260811 11:18:38.999475 12308 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dccede190e2437b8bcef1332b5e169d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:38 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:48478
I20260811 11:18:38.999488 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 11930
I20260811 11:18:39.002838 12428 tablet_replica.cc:330] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: stopping tablet replica
I20260811 11:18:39.003120 12428 raft_consensus.cc:2229] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:18:39.003304 12428 raft_consensus.cc:2258] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:18:39.003937 12428 ts_tablet_manager.cc:1905] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:18:39.006706 12428 ts_tablet_manager.cc:1918] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:18:39.006844 12428 log.cc:1196] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/2dccede190e2437b8bcef1332b5e169d
I20260811 11:18:39.007200 12428 ts_tablet_manager.cc:1939] T 2dccede190e2437b8bcef1332b5e169d P 218c9e30682044c3bab931618da129cb: Deleting consensus metadata
I20260811 11:18:39.007776 11847 catalog_manager.cc:4841] TS 218c9e30682044c3bab931618da129cb (127.28.25.3:44201): tablet 2dccede190e2437b8bcef1332b5e169d (table test-table [id=28a2cc87e9ef4e2ca51728d85ea7bb7c]) successfully deleted
I20260811 11:18:39.009912 12426 tablet_replica.cc:330] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: stopping tablet replica
I20260811 11:18:39.010203 12426 raft_consensus.cc:2229] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:18:39.010373 12426 raft_consensus.cc:2258] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:18:39.011106 12426 ts_tablet_manager.cc:1905] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:18:39.013335 12426 ts_tablet_manager.cc:1918] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:18:39.013549 12426 log.cc:1196] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/2dccede190e2437b8bcef1332b5e169d
I20260811 11:18:39.013989 12426 ts_tablet_manager.cc:1939] T 2dccede190e2437b8bcef1332b5e169d P e9ffae368ba74ac8b9245c80111586e3: Deleting consensus metadata
I20260811 11:18:39.014628 11849 catalog_manager.cc:4841] TS e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589): tablet 2dccede190e2437b8bcef1332b5e169d (table test-table [id=28a2cc87e9ef4e2ca51728d85ea7bb7c]) successfully deleted
W20260811 11:18:39.015004 11849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:36293 (error 108)
I20260811 11:18:39.015298 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 12084
I20260811 11:18:39.015856 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.015491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36293 (local address 127.0.0.1:34090)
0811 11:18:39.015550 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:39.015582 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 11:18:39.019812 11849 catalog_manager.cc:4584] TS e93aad5aa4844bc08cb3d5d64a4b262f (127.28.25.1:36293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2dccede190e2437b8bcef1332b5e169d: Network error: Client connection negotiation failed: client connection to 127.28.25.1:36293: connect: Connection refused (error 111)
I20260811 11:18:39.024760 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 12235
I20260811 11:18:39.033598 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 11822
I20260811 11:18:39.042654 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39205
--webserver_interface=127.28.25.62
--webserver_port=35295
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:39205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:39.135643 12430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:39.135953 12430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:39.135999 12430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:39.137317 12430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:18:39.137410 12430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:39.137439 12430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:39.137485 12430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:18:39.137517 12430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:18:39.139595 12430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:39205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=35295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:39.140030 12430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:39.140309 12430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:39.151154 12438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:39.151207 12430 server_base.cc:1034] running on GCE node
W20260811 11:18:39.151173 12439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:39.151154 12441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:39.151691 12430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:39.152081 12430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:39.153272 12430 hybrid_clock.cc:648] HybridClock initialized: now 1786447119153247 us; error 44 us; skew 500 ppm
Aug 11 11:18:39 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447119, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:39.156450 12430 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:18:39.157038 12430 webserver.cc:460] Webserver started at http://127.28.25.62:35295/ using document root <none> and password file <none>
I20260811 11:18:39.157310 12430 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:39.157424 12430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:39.159016 12430 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:18:39.159909 12449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:39.160286 12430 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:39.160440 12430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "adb8ab01dc854a73894c9921aa6f10a0"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "7a25e13f0555cc0b9c10f6d24cf3b925"
server_key_iv: "d95c5a7b68500465d3f7a000ce79c514"
server_key_version: "encryptionkey@0"
I20260811 11:18:39.160957 12430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:39.181432 12430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:39.184135 12430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:39.184370 12430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:39.189646 12430 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:39205
I20260811 11:18:39.189703 12505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:39205 every 8 connection(s)
I20260811 11:18:39.190131 12430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:18:39.190593 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 12430
I20260811 11:18:39.191039 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:36293
--local_ip_for_outbound_sockets=127.28.25.1
--tserver_master_addrs=127.28.25.62:39205
--webserver_port=36543
--webserver_interface=127.28.25.1
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:18:39.192209 12506 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:18:39.192914 12506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Bootstrap starting.
I20260811 11:18:39.205665 12506 log.cc:824] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:39.208424 12506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:18:39.208819 12506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Bootstrap complete.
I20260811 11:18:39.212375 12506 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:39.212740 12506 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb8ab01dc854a73894c9921aa6f10a0, State: Initialized, Role: FOLLOWER
I20260811 11:18:39.212917 12506 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:39.213018 12506 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:18:39.213074 12506 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:18:39.213128 12506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 1 FOLLOWER]: Advancing to term 2
I20260811 11:18:39.214704 12506 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:39.214864 12506 leader_election.cc:302] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb8ab01dc854a73894c9921aa6f10a0; no voters:
I20260811 11:18:39.215076 12506 leader_election.cc:288] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 11:18:39.215232 12510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 11:18:39.215343 12506 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:18:39.215435 12510 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 2 LEADER]: Becoming Leader. State: Replica: adb8ab01dc854a73894c9921aa6f10a0, State: Running, Role: LEADER
I20260811 11:18:39.215713 12510 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:39.216065 12511 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "adb8ab01dc854a73894c9921aa6f10a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } } }
I20260811 11:18:39.216161 12511 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:39.216339 12512 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb8ab01dc854a73894c9921aa6f10a0. Latest consensus state: current_term: 2 leader_uuid: "adb8ab01dc854a73894c9921aa6f10a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } } }
I20260811 11:18:39.216413 12512 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:39.216768 12519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:18:39.217816 12519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:18:39.217936 12519 catalog_manager.cc:1259] Loaded cluster ID: 8b94a3019e214f569f948f4b5f88a6e9
I20260811 11:18:39.217972 12519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:18:39.218225 12519 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260811 11:18:39.222908 12519 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260811 11:18:39.225958 12519 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:18:39.226224 12519 catalog_manager.cc:5879] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Loaded TSK: 0
I20260811 11:18:39.226475 12519 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 11:18:39.288491 12508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:39.288743 12508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:39.288786 12508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:39.290081 12508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:39.290158 12508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:39.290208 12508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:18:39.291935 12508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:36293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=36543
--tserver_master_addrs=127.28.25.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:39.292239 12508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:39.292527 12508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:39.298817 12535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:39.299003 12538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:39.298913 12536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:39.299292 12508 server_base.cc:1034] running on GCE node
I20260811 11:18:39.299542 12508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:39.299820 12508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:39.301007 12508 hybrid_clock.cc:648] HybridClock initialized: now 1786447119300980 us; error 42 us; skew 500 ppm
Aug 11 11:18:39 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447119, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:39.304347 12508 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:18:39.304927 12508 webserver.cc:460] Webserver started at http://127.28.25.1:36543/ using document root <none> and password file <none>
I20260811 11:18:39.305198 12508 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:39.305335 12508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:39.306788 12508 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 11:18:39.307610 12545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:39.307885 12508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:39.308000 12508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "e93aad5aa4844bc08cb3d5d64a4b262f"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "ec44e00ff57330148b8c8b9331f3804d"
server_key_iv: "15276159fde0a2117433fcadfc65538e"
server_key_version: "encryptionkey@0"
I20260811 11:18:39.308486 12508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:39.330709 12508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:39.333611 12508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:39.333793 12508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:39.334076 12508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:39.334650 12553 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 11:18:39.336396 12508 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 11:18:39.336462 12508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260811 11:18:39.336501 12508 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 11:18:39.337373 12508 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 11:18:39.337423 12508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260811 11:18:39.337512 12553 tablet_bootstrap.cc:492] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Bootstrap starting.
I20260811 11:18:39.345026 12508 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:36293
I20260811 11:18:39.345320 12670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:36293 every 8 connection(s)
I20260811 11:18:39.345530 12508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:18:39.348843 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 12508
I20260811 11:18:39.349164 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:43589
--local_ip_for_outbound_sockets=127.28.25.2
--tserver_master_addrs=127.28.25.62:39205
--webserver_port=34297
--webserver_interface=127.28.25.2
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:18:39.350039 12553 log.cc:824] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:39.352846 12553 tablet_bootstrap.cc:492] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:18:39.353178 12553 tablet_bootstrap.cc:492] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Bootstrap complete.
I20260811 11:18:39.354557 12553 ts_tablet_manager.cc:1397] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
Aug 11 11:18:39 dist-test-slave-2w3w krb5kdc[11809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447119, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:39.356470 12553 raft_consensus.cc:348] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.357790 12553 raft_consensus.cc:729] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93aad5aa4844bc08cb3d5d64a4b262f, State: Initialized, Role: FOLLOWER
I20260811 11:18:39.357988 12553 consensus_queue.cc:260] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.358327 12553 ts_tablet_manager.cc:1428] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 11:18:39.362736 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.346293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:55289)
0811 11:18:39.348399 (+ 2106us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:39.348460 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:39.349787 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:39.350540 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:39.350544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:39.350915 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:39.351222 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.351230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.352888 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.352890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.354141 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.354145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.355153 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.355155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.355299 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.355940 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:39.355950 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:39.358022 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:39.360794 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:39.360803 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:39.360805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:39.361136 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:39.361450 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:39.361451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:39.361451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:39.361582 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:39.362059 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:39.362067 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:39.362225 (+ 158us) client_negotiation.cc:769] Sending connection context
0811 11:18:39.362327 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 11:18:39.362520 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2046,"thread_start_us":110,"threads_started":1}
I20260811 11:18:39.363505 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.346404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:55289 (local address 127.28.25.62:39205)
0811 11:18:39.349085 (+ 2681us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.349096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.349158 (+ 62us) server_negotiation.cc:407] Connection header received
0811 11:18:39.350103 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.350161 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.350235 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.350397 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:39.351373 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.352749 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.354259 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.354880 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.355030 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.358230 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:39.358234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:39.358235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:39.358248 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:39.360661 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:39.361237 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:39.361240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:39.361241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:39.361375 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:39.361650 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:39.361650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:39.361651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:39.361944 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:39.362026 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.363151 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.363301 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2619,"thread_start_us":104,"threads_started":1}
I20260811 11:18:39.364254 12671 heartbeater.cc:347] Connected to a master server at 127.28.25.62:39205
I20260811 11:18:39.364364 12671 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:39.364564 12671 heartbeater.cc:511] Master 127.28.25.62:39205 requested a full tablet report, sending...
I20260811 11:18:39.365494 12468 ts_manager.cc:194] Registered new tserver with Master: e93aad5aa4844bc08cb3d5d64a4b262f (127.28.25.1:36293)
I20260811 11:18:39.366720 12468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:55289
I20260811 11:18:39.367753 12671 heartbeater.cc:503] Master 127.28.25.62:39205 was elected leader, sending a full tablet report...
Aug 11 11:18:39 dist-test-slave-2w3w krb5kdc[11809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447119, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.1@KRBTEST.COM
I20260811 11:18:39.377045 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.366428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.28.25.1:36293)
0811 11:18:39.366843 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.366847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.366857 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:18:39.366890 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.366892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.366905 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.368267 (+ 1362us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:39.368972 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.369884 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.371181 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.371650 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.371729 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.374105 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:39.374114 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:39.374114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:39.374121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:39.375806 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:39.376230 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:39.376232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:39.376233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:39.376286 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:39.376481 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:39.376482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:39.376482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:39.376612 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:39.376675 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.376895 (+ 220us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.376971 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":75,"threads_started":1}
I20260811 11:18:39.377076 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.366146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36293 (local address 127.0.0.1:34106)
0811 11:18:39.366597 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:39.366636 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:39.366702 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:39.368336 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:39.368340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:39.368688 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:39.368863 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.368866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.369984 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.369985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.371095 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.371099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.371739 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.371740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.371826 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.372391 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:39.372397 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:39.374011 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:39.375887 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:39.375895 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:39.375897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:39.376147 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:39.376352 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:39.376352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:39.376353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:39.376419 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:39.376677 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:39.376684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:39.376756 (+ 72us) client_negotiation.cc:769] Sending connection context
0811 11:18:39.376844 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 11:18:39.376978 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":105,"threads_started":1}
I20260811 11:18:39.377501 12600 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dccede190e2437b8bcef1332b5e169d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:38 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:34106
I20260811 11:18:39.378125 12680 tablet_replica.cc:330] stopping tablet replica
I20260811 11:18:39.378239 12680 raft_consensus.cc:2229] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:18:39.378314 12680 raft_consensus.cc:2258] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:18:39.378654 12680 ts_tablet_manager.cc:1905] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:18:39.380791 12680 ts_tablet_manager.cc:1918] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:18:39.380929 12680 log.cc:1196] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/2dccede190e2437b8bcef1332b5e169d
I20260811 11:18:39.381182 12680 ts_tablet_manager.cc:1939] T 2dccede190e2437b8bcef1332b5e169d P e93aad5aa4844bc08cb3d5d64a4b262f: Deleting consensus metadata
I20260811 11:18:39.381742 12453 catalog_manager.cc:4841] TS e93aad5aa4844bc08cb3d5d64a4b262f (127.28.25.1:36293): tablet 2dccede190e2437b8bcef1332b5e169d (table test-table [id=28a2cc87e9ef4e2ca51728d85ea7bb7c]) successfully deleted
W20260811 11:18:39.444339 12676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:39.444571 12676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:39.444598 12676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:39.445757 12676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:39.445796 12676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:39.445823 12676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:18:39.447412 12676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:43589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=34297
--tserver_master_addrs=127.28.25.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:39.447623 12676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:39.447834 12676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:39.453408 12690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:39.453411 12687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:39.453411 12686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:39.453430 12676 server_base.cc:1034] running on GCE node
I20260811 11:18:39.453881 12676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:39.454105 12676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:39.455268 12676 hybrid_clock.cc:648] HybridClock initialized: now 1786447119455241 us; error 41 us; skew 500 ppm
Aug 11 11:18:39 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447119, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:39.458026 12676 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:18:39.458539 12676 webserver.cc:460] Webserver started at http://127.28.25.2:34297/ using document root <none> and password file <none>
I20260811 11:18:39.458806 12676 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:39.458896 12676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:39.460503 12676 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:39.461227 12697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:39.461465 12676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:39.461580 12676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "e9ffae368ba74ac8b9245c80111586e3"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "f0c0e85aae0d4a681a26a7307ac6f880"
server_key_iv: "ce067474f6650d10cf9467d353443a06"
server_key_version: "encryptionkey@0"
I20260811 11:18:39.461966 12676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:39.474145 12676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:39.476817 12676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:39.477003 12676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:39.477283 12676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:39.477617 12676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:39.477674 12676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:39.477722 12676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:39.477747 12676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:39.484480 12676 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:43589
I20260811 11:18:39.484529 12826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:43589 every 8 connection(s)
I20260811 11:18:39.484915 12676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:18:39.487375 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 12676
I20260811 11:18:39.487720 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:44201
--local_ip_for_outbound_sockets=127.28.25.3
--tserver_master_addrs=127.28.25.62:39205
--webserver_port=44765
--webserver_interface=127.28.25.3
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:18:39 dist-test-slave-2w3w krb5kdc[11809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447119, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:39.496536 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.485699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:46171 (local address 127.28.25.62:39205)
0811 11:18:39.485832 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.485834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.486081 (+ 247us) server_negotiation.cc:407] Connection header received
0811 11:18:39.486800 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.486801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.486817 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.486910 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:39.488450 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.488980 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.490396 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.490617 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.490701 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.493287 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:39.493289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:39.493289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:39.493297 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:39.494924 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:39.495460 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:39.495461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:39.495461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:39.495527 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:39.495806 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:39.495807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:39.495808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:39.495937 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:39.496016 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.496334 (+ 318us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.496438 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260811 11:18:39.496738 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.485696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:46171)
0811 11:18:39.485935 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:39.485981 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:39.486675 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:39.487030 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:39.487033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:39.487997 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:39.488296 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.488303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.489103 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.489104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.490271 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.490274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.490712 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.490714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.490852 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.491544 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:39.491553 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:39.493136 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:39.495068 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:39.495077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:39.495080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:39.495368 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:39.495603 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:39.495604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:39.495604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:39.495725 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:39.496017 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:39.496026 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:39.496183 (+ 157us) client_negotiation.cc:769] Sending connection context
0811 11:18:39.496300 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 11:18:39.496481 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
I20260811 11:18:39.497376 12827 heartbeater.cc:347] Connected to a master server at 127.28.25.62:39205
I20260811 11:18:39.497483 12827 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:39.497704 12827 heartbeater.cc:511] Master 127.28.25.62:39205 requested a full tablet report, sending...
I20260811 11:18:39.498224 12468 ts_manager.cc:194] Registered new tserver with Master: e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589)
I20260811 11:18:39.498816 12468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:46171
W20260811 11:18:39.576494 12830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:39.576732 12830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:39.576761 12830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:39.577965 12830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:39.578011 12830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:39.578042 12830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:18:39.579591 12830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:44201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=44765
--tserver_master_addrs=127.28.25.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:39.579810 12830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:39.580019 12830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:39.585924 12836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:39.586051 12835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:39.585924 12838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:39.586249 12830 server_base.cc:1034] running on GCE node
I20260811 11:18:39.586488 12830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:39.586779 12830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:39.587949 12830 hybrid_clock.cc:648] HybridClock initialized: now 1786447119587924 us; error 42 us; skew 500 ppm
Aug 11 11:18:39 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447119, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:39.590613 12830 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:18:39.591150 12830 webserver.cc:460] Webserver started at http://127.28.25.3:44765/ using document root <none> and password file <none>
I20260811 11:18:39.591375 12830 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:39.591439 12830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:39.592926 12830 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:39.593698 12845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:39.593961 12830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:39.594065 12830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "218c9e30682044c3bab931618da129cb"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "b9764fa3f7311338083796b2bde883dc"
server_key_iv: "8941f156f64e94ef96a15ffe1555c074"
server_key_version: "encryptionkey@0"
I20260811 11:18:39.594461 12830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:39.615656 12830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:39.618281 12830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:39.618482 12830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:39.618729 12830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:39.619107 12830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:39.619143 12830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:39.619166 12830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:39.619211 12830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:39.625590 12830 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:44201
I20260811 11:18:39.625682 12971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:44201 every 8 connection(s)
I20260811 11:18:39.626056 12830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
Aug 11 11:18:39 dist-test-slave-2w3w krb5kdc[11809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447119, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:39.635285 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 12830
I20260811 11:18:39.635762 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.626871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:54301 (local address 127.28.25.62:39205)
0811 11:18:39.627014 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.627016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.627168 (+ 152us) server_negotiation.cc:407] Connection header received
0811 11:18:39.627684 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.627685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.627695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.627762 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:39.628917 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.629405 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.630464 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.630664 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.630734 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.632987 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:39.632988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:39.632989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:39.632996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:39.634215 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:39.634608 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:39.634609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:39.634610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:39.634661 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:39.634882 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:39.634883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:39.634883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:39.635051 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:39.635106 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.635557 (+ 451us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.635650 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 11:18:39.635963 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.626840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:54301)
0811 11:18:39.627045 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:39.627091 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:39.627596 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:39.627827 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:39.627831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:39.628456 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:39.628668 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.628673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.629505 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.629506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.630340 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.630343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.630749 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.630750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.630875 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.631492 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:39.631501 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:39.632896 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:39.634291 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:39.634298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:39.634300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:39.634540 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:39.634728 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:39.634729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:39.634729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:39.634828 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:39.635118 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:39.635126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:39.635308 (+ 182us) client_negotiation.cc:769] Sending connection context
0811 11:18:39.635502 (+ 194us) client_negotiation.cc:240] Negotiation successful
0811 11:18:39.635686 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260811 11:18:39.636562 12972 heartbeater.cc:347] Connected to a master server at 127.28.25.62:39205
I20260811 11:18:39.636686 12972 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:39.636941 12972 heartbeater.cc:511] Master 127.28.25.62:39205 requested a full tablet report, sending...
I20260811 11:18:39.637359 12468 ts_manager.cc:194] Registered new tserver with Master: 218c9e30682044c3bab931618da129cb (127.28.25.3:44201)
I20260811 11:18:39.637795 12468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:54301
I20260811 11:18:39.642359 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.635742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.28.25.62:39205)
0811 11:18:39.635862 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.635864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.635986 (+ 122us) server_negotiation.cc:407] Connection header received
0811 11:18:39.636117 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.636118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.636130 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.636187 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:39.636945 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.637644 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.638576 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.638797 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.638857 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.640136 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:39.640137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:39.640137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:39.640143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:39.641359 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:39.641690 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:39.641691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:39.641691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:39.641742 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:39.641923 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:39.641924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:39.641924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:39.642027 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:39.642154 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.642202 (+ 48us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.642260 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":53,"threads_started":1}
I20260811 11:18:39.643244 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:18:39.646154 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.643598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.28.25.62:39205)
0811 11:18:39.643669 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.643671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.643769 (+ 98us) server_negotiation.cc:407] Connection header received
0811 11:18:39.643843 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.643843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.643925 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.644002 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:18:39.644128 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.644608 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.645366 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.645575 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.645661 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.645831 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:18:39.645943 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:18:39.646034 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.646058 (+ 24us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.646076 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:18:39.646450 12468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:18:39.648578 12468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:18:39.651233 12600 tablet_service.cc:1467] Processing CreateTablet for tablet 388dfddea44743eca52461ed3acd1785 (DEFAULT_TABLE table=test-table [id=2e0808ac8916434eb0fbb67fde21e7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:39.651484 12600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388dfddea44743eca52461ed3acd1785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:39.653124 12553 tablet_bootstrap.cc:492] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Bootstrap starting.
I20260811 11:18:39.653488 12553 tablet_bootstrap.cc:654] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:39.654094 12553 tablet_bootstrap.cc:492] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: No bootstrap required, opened a new log
I20260811 11:18:39.654152 12553 ts_tablet_manager.cc:1397] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260811 11:18:39.654505 12553 raft_consensus.cc:348] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.654590 12553 raft_consensus.cc:374] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:39.654640 12553 raft_consensus.cc:729] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93aad5aa4844bc08cb3d5d64a4b262f, State: Initialized, Role: FOLLOWER
I20260811 11:18:39.654690 12553 consensus_queue.cc:260] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.654829 12553 ts_tablet_manager.cc:1428] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 11:18:39.654896 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.650700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.28.25.3:44201)
0811 11:18:39.651129 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.651134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.651144 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:18:39.651186 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.651188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.651206 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.651294 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:39.651508 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.652588 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.653826 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.654476 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.654567 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.654631 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.654766 (+ 135us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.654806 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":112,"threads_started":1}
I20260811 11:18:39.654886 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.650602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:44201 (local address 127.0.0.1:48480)
0811 11:18:39.650680 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:39.650694 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:39.650740 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:39.651304 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:39.651305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:39.651307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:39.651363 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.651365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.652707 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.652708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.653731 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.653734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.654565 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.654567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.654639 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.654668 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 11:18:39.654749 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 11:18:39.654804 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 11:18:39.655089 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.650680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:43589 (local address 127.0.0.1:38420)
0811 11:18:39.650872 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:39.650881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:39.650943 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:39.651458 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:39.651459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:39.651461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:39.651495 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.651496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.652399 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.652400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.653582 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.653585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.654830 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.654831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.654909 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.654914 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:18:39.654951 (+ 37us) client_negotiation.cc:240] Negotiation successful
0811 11:18:39.655017 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":147,"threads_started":1}
I20260811 11:18:39.655558 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.651090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.28.25.2:43589)
0811 11:18:39.651262 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.651267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.651278 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:18:39.651317 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.651319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.651334 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.651465 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:39.651545 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.652280 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.653692 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.654698 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.655260 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.655338 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.655367 (+ 29us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.655432 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260811 11:18:39.655684 12898 tablet_service.cc:1467] Processing CreateTablet for tablet 388dfddea44743eca52461ed3acd1785 (DEFAULT_TABLE table=test-table [id=2e0808ac8916434eb0fbb67fde21e7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:39.656085 12898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388dfddea44743eca52461ed3acd1785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:39.656268 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 388dfddea44743eca52461ed3acd1785 (DEFAULT_TABLE table=test-table [id=2e0808ac8916434eb0fbb67fde21e7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:39.656589 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388dfddea44743eca52461ed3acd1785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:39.659919 12982 tablet_bootstrap.cc:492] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: Bootstrap starting.
I20260811 11:18:39.660331 12983 tablet_bootstrap.cc:492] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: Bootstrap starting.
I20260811 11:18:39.660811 12982 tablet_bootstrap.cc:654] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:39.661067 12983 tablet_bootstrap.cc:654] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:39.661509 12982 log.cc:824] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:39.661651 12983 log.cc:824] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:39.662344 12982 tablet_bootstrap.cc:492] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: No bootstrap required, opened a new log
I20260811 11:18:39.662386 12983 tablet_bootstrap.cc:492] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: No bootstrap required, opened a new log
I20260811 11:18:39.662459 12982 ts_tablet_manager.cc:1397] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:18:39.662446 12983 ts_tablet_manager.cc:1397] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:18:39.664214 12983 raft_consensus.cc:348] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.664354 12983 raft_consensus.cc:374] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:39.664376 12983 raft_consensus.cc:729] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ffae368ba74ac8b9245c80111586e3, State: Initialized, Role: FOLLOWER
I20260811 11:18:39.664557 12983 consensus_queue.cc:260] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.664474 12982 raft_consensus.cc:348] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.664647 12982 raft_consensus.cc:374] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:39.664696 12982 raft_consensus.cc:729] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218c9e30682044c3bab931618da129cb, State: Initialized, Role: FOLLOWER
I20260811 11:18:39.664819 12982 consensus_queue.cc:260] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.664915 12827 heartbeater.cc:503] Master 127.28.25.62:39205 was elected leader, sending a full tablet report...
I20260811 11:18:39.664897 12983 ts_tablet_manager.cc:1428] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 11:18:39.665115 12982 ts_tablet_manager.cc:1428] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:18:39.665158 12972 heartbeater.cc:503] Master 127.28.25.62:39205 was elected leader, sending a full tablet report...
I20260811 11:18:39.713425 12677 raft_consensus.cc:482] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:39.713775 12677 raft_consensus.cc:504] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.714265 12677 leader_election.cc:288] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218c9e30682044c3bab931618da129cb (127.28.25.3:44201), e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589)
I20260811 11:18:39.717985 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.714451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:44201 (local address 127.28.25.1:38375)
0811 11:18:39.714689 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:39.714698 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:39.714782 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:39.715056 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:39.715057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:39.715059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:39.715136 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.715138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.716024 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.716026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.716962 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.716965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.717660 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.717662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.717746 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.717770 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:18:39.717844 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 11:18:39.717890 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":16,"thread_start_us":130,"threads_started":1}
I20260811 11:18:39.717971 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.714512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:38375 (local address 127.28.25.3:44201)
0811 11:18:39.714606 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.714609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.714868 (+ 259us) server_negotiation.cc:407] Connection header received
0811 11:18:39.714929 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.714930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.714947 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.715037 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:39.715213 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.715902 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.717058 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.717540 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.717636 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.717653 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.717850 (+ 197us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.717898 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 11:18:39.718114 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.714531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:46627 (local address 127.28.25.2:43589)
0811 11:18:39.714587 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.714589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.714603 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:18:39.714669 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.714669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.714680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.714754 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:39.715023 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.715819 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.717183 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.717684 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.717785 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.717797 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.717995 (+ 198us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.718040 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 11:18:39.718114 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.714444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:43589 (local address 127.28.25.1:46627)
0811 11:18:39.714504 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:39.714518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:39.714575 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:39.714779 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:39.714780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:39.714782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:39.714867 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.714869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.715932 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.715933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.717059 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:39.717062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.717784 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.717785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:39.717893 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.717899 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:18:39.717986 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 11:18:39.718040 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260811 11:18:39.718683 12775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388dfddea44743eca52461ed3acd1785" candidate_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9ffae368ba74ac8b9245c80111586e3" is_pre_election: true
I20260811 11:18:39.718880 12775 raft_consensus.cc:2454] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e93aad5aa4844bc08cb3d5d64a4b262f in term 0.
I20260811 11:18:39.718818 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388dfddea44743eca52461ed3acd1785" candidate_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218c9e30682044c3bab931618da129cb" is_pre_election: true
I20260811 11:18:39.719049 12921 raft_consensus.cc:2454] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e93aad5aa4844bc08cb3d5d64a4b262f in term 0.
I20260811 11:18:39.719197 12549 leader_election.cc:302] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e93aad5aa4844bc08cb3d5d64a4b262f, e9ffae368ba74ac8b9245c80111586e3; no voters:
I20260811 11:18:39.719372 12677 raft_consensus.cc:2781] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:18:39.719439 12677 raft_consensus.cc:482] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:18:39.719465 12677 raft_consensus.cc:3037] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:39.720268 12677 raft_consensus.cc:504] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.720443 12677 leader_election.cc:288] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 1 election: Requested vote from peers 218c9e30682044c3bab931618da129cb (127.28.25.3:44201), e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589)
I20260811 11:18:39.720646 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388dfddea44743eca52461ed3acd1785" candidate_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218c9e30682044c3bab931618da129cb"
I20260811 11:18:39.720662 12775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388dfddea44743eca52461ed3acd1785" candidate_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9ffae368ba74ac8b9245c80111586e3"
I20260811 11:18:39.720744 12921 raft_consensus.cc:3037] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:39.720765 12775 raft_consensus.cc:3037] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:39.721688 12775 raft_consensus.cc:2454] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e93aad5aa4844bc08cb3d5d64a4b262f in term 1.
I20260811 11:18:39.721728 12921 raft_consensus.cc:2454] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e93aad5aa4844bc08cb3d5d64a4b262f in term 1.
I20260811 11:18:39.721937 12549 leader_election.cc:302] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e93aad5aa4844bc08cb3d5d64a4b262f, e9ffae368ba74ac8b9245c80111586e3; no voters:
I20260811 11:18:39.722095 12677 raft_consensus.cc:2781] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:18:39.722307 12677 raft_consensus.cc:686] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 LEADER]: Becoming Leader. State: Replica: e93aad5aa4844bc08cb3d5d64a4b262f, State: Running, Role: LEADER
I20260811 11:18:39.722409 12677 consensus_queue.cc:237] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:39.723403 12468 catalog_manager.cc:5495] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f reported cstate change: term changed from 0 to 1, leader changed from <none> to e93aad5aa4844bc08cb3d5d64a4b262f (127.28.25.1). New cstate: current_term: 1 leader_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } health_report { overall_health: HEALTHY } } }
I20260811 11:18:39.734318 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.731669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.28.25.1:36293)
0811 11:18:39.731745 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:39.731748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:39.731758 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:18:39.731805 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:39.731806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:39.731825 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:39.731926 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:18:39.732066 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.732623 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.733436 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:39.733671 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:39.733733 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:39.733956 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:18:39.734074 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:18:39.734134 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:39.734226 (+ 92us) server_negotiation.cc:299] Negotiation successful
0811 11:18:39.734250 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260811 11:18:39.735519 12828 tablet.cc:2369] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:18:39.736379 12775 raft_consensus.cc:1261] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3 [term 1 FOLLOWER]: Refusing update from remote peer e93aad5aa4844bc08cb3d5d64a4b262f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:18:39.736377 12921 raft_consensus.cc:1261] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb [term 1 FOLLOWER]: Refusing update from remote peer e93aad5aa4844bc08cb3d5d64a4b262f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:18:39.736758 12992 consensus_queue.cc:1035] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:18:39.736939 12677 consensus_queue.cc:1035] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:18:39.739729 12996 mvcc.cc:205] Tried to move back new op lower bound from 7317287402437017600 to 7317287402383765504. Current Snapshot: MvccSnapshot[applied={T|T < 7317287402437017600}]
I20260811 11:18:39.740531 12997 mvcc.cc:205] Tried to move back new op lower bound from 7317287402437017600 to 7317287402383765504. Current Snapshot: MvccSnapshot[applied={T|T < 7317287402437017600}]
I20260811 11:18:39.741816 12467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:18:39.741917 12467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:18:39.742717 12467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Sending DeleteTablet for 3 replicas of tablet 388dfddea44743eca52461ed3acd1785
I20260811 11:18:39.743062 12898 tablet_service.cc:1514] Processing DeleteTablet for tablet 388dfddea44743eca52461ed3acd1785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:39 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:48480
I20260811 11:18:39.743136 12600 tablet_service.cc:1514] Processing DeleteTablet for tablet 388dfddea44743eca52461ed3acd1785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:39 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:34106
I20260811 11:18:39.743129 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 388dfddea44743eca52461ed3acd1785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:39 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:38420
I20260811 11:18:39.743228 12680 tablet_replica.cc:330] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: stopping tablet replica
I20260811 11:18:39.743283 12680 raft_consensus.cc:2229] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:18:39.743373 12680 raft_consensus.cc:2258] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:18:39.743520 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 12508
I20260811 11:18:39.743741 12680 ts_tablet_manager.cc:1905] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:18:39.744148 13005 tablet_replica.cc:330] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: stopping tablet replica
I20260811 11:18:39.744253 13005 raft_consensus.cc:2229] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:18:39.744361 13005 raft_consensus.cc:2258] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:18:39.745039 13005 ts_tablet_manager.cc:1905] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:18:39.747120 13005 ts_tablet_manager.cc:1918] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:18:39.747223 13005 log.cc:1196] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/388dfddea44743eca52461ed3acd1785
I20260811 11:18:39.747560 13005 ts_tablet_manager.cc:1939] T 388dfddea44743eca52461ed3acd1785 P 218c9e30682044c3bab931618da129cb: Deleting consensus metadata
I20260811 11:18:39.748163 12451 catalog_manager.cc:4841] TS 218c9e30682044c3bab931618da129cb (127.28.25.3:44201): tablet 388dfddea44743eca52461ed3acd1785 (table test-table [id=2e0808ac8916434eb0fbb67fde21e7b8]) successfully deleted
I20260811 11:18:39.748687 13006 tablet_replica.cc:330] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: stopping tablet replica
I20260811 11:18:39.748833 13006 raft_consensus.cc:2229] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:18:39.748966 13006 raft_consensus.cc:2258] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:18:39.749593 13006 ts_tablet_manager.cc:1905] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:18:39.751881 13006 ts_tablet_manager.cc:1918] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:18:39.751982 13006 log.cc:1196] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/388dfddea44743eca52461ed3acd1785
I20260811 11:18:39.752303 13006 ts_tablet_manager.cc:1939] T 388dfddea44743eca52461ed3acd1785 P e9ffae368ba74ac8b9245c80111586e3: Deleting consensus metadata
I20260811 11:18:39.752883 12453 catalog_manager.cc:4841] TS e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589): tablet 388dfddea44743eca52461ed3acd1785 (table test-table [id=2e0808ac8916434eb0fbb67fde21e7b8]) successfully deleted
W20260811 11:18:39.754943 12453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:36293 (error 108)
I20260811 11:18:39.755205 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 12676
I20260811 11:18:39.758903 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:39.757366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36293 (local address 127.0.0.1:34128)
0811 11:18:39.758787 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:39.758851 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1388}
W20260811 11:18:39.759126 12453 catalog_manager.cc:4584] TS e93aad5aa4844bc08cb3d5d64a4b262f (127.28.25.1:36293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 388dfddea44743eca52461ed3acd1785: Network error: Client connection negotiation failed: client connection to 127.28.25.1:36293: connect: Connection refused (error 111)
I20260811 11:18:39.765151 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 12830
I20260811 11:18:39.772449 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 12430
I20260811 11:18:39.781725 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39205
--webserver_interface=127.28.25.62
--webserver_port=35295
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:39205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:39.872390 13008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:39.872680 13008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:39.872721 13008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:39.873970 13008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:18:39.874043 13008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:39.874068 13008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:39.874100 13008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:18:39.874132 13008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:18:39.875800 13008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:39205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=35295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:39.876132 13008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:39.876394 13008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:39.881844 13015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:39.882020 13008 server_base.cc:1034] running on GCE node
W20260811 11:18:39.881992 13017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:39.881852 13014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:39.882407 13008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:39.882700 13008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:39.884023 13008 hybrid_clock.cc:648] HybridClock initialized: now 1786447119883986 us; error 54 us; skew 500 ppm
Aug 11 11:18:39 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447119, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:39.886909 13008 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:18:39.887557 13008 webserver.cc:460] Webserver started at http://127.28.25.62:35295/ using document root <none> and password file <none>
I20260811 11:18:39.887813 13008 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:39.887862 13008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:39.889290 13008 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:39.890012 13025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:39.890297 13008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:39.890429 13008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "adb8ab01dc854a73894c9921aa6f10a0"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "7a25e13f0555cc0b9c10f6d24cf3b925"
server_key_iv: "d95c5a7b68500465d3f7a000ce79c514"
server_key_version: "encryptionkey@0"
I20260811 11:18:39.890815 13008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:39.900127 13008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:39.902359 13008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:39.902549 13008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:39.907478 13008 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:39205
I20260811 11:18:39.907531 13084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:39205 every 8 connection(s)
I20260811 11:18:39.907971 13008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:18:39.908265 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 13008
I20260811 11:18:39.908588 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:36293
--local_ip_for_outbound_sockets=127.28.25.1
--tserver_master_addrs=127.28.25.62:39205
--webserver_port=36543
--webserver_interface=127.28.25.1
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:18:39.909766 13085 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:18:39.910658 13085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Bootstrap starting.
I20260811 11:18:39.923794 13085 log.cc:824] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:39.926321 13085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:18:39.926677 13085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0: Bootstrap complete.
I20260811 11:18:39.929287 13085 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:39.929580 13085 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb8ab01dc854a73894c9921aa6f10a0, State: Initialized, Role: FOLLOWER
I20260811 11:18:39.929800 13085 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:39.929906 13085 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:18:39.929963 13085 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:18:39.930012 13085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 2 FOLLOWER]: Advancing to term 3
I20260811 11:18:39.931176 13085 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:39.931306 13085 leader_election.cc:302] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb8ab01dc854a73894c9921aa6f10a0; no voters:
I20260811 11:18:39.931536 13085 leader_election.cc:288] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 11:18:39.931578 13089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 11:18:39.931696 13089 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [term 3 LEADER]: Becoming Leader. State: Replica: adb8ab01dc854a73894c9921aa6f10a0, State: Running, Role: LEADER
I20260811 11:18:39.931800 13089 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } }
I20260811 11:18:39.931838 13085 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:18:39.932215 13091 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb8ab01dc854a73894c9921aa6f10a0. Latest consensus state: current_term: 3 leader_uuid: "adb8ab01dc854a73894c9921aa6f10a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } } }
I20260811 11:18:39.932243 13090 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "adb8ab01dc854a73894c9921aa6f10a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8ab01dc854a73894c9921aa6f10a0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39205 } } }
I20260811 11:18:39.932415 13090 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:39.932647 13091 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8ab01dc854a73894c9921aa6f10a0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:39.932657 13097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:18:39.933638 13097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:18:39.933769 13097 catalog_manager.cc:1259] Loaded cluster ID: 8b94a3019e214f569f948f4b5f88a6e9
I20260811 11:18:39.933818 13097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260811 11:18:39.934247 13097 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f38e272e9ad google::LogMessage::Fail() at ??:0
@ 0x7f38e273188a google::LogMessage::SendToLog() at ??:0
@ 0x7f38e272e4de google::LogMessage::Flush() at ??:0
@ 0x7f38e272fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f38ea64f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f38ea6588b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f38e45fc0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f38e45f1954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f38e54556db start_thread at ??:0
@ 0x7f38e122c61f clone at ??:0
W20260811 11:18:40.013621 13087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:40.013921 13087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:40.013971 13087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:40.015573 13087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:40.015648 13087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:40.015702 13087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:18:40.017529 13087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:36293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=36543
--tserver_master_addrs=127.28.25.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:40.017863 13087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:40.018157 13087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:40.024034 13124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:40.024039 13125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:40.024498 13087 server_base.cc:1034] running on GCE node
W20260811 11:18:40.024693 13128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:40.024951 13087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:40.025321 13087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:40.026539 13087 hybrid_clock.cc:648] HybridClock initialized: now 1786447120026522 us; error 35 us; skew 500 ppm
Aug 11 11:18:40 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447120, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:40.030153 13087 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:18:40.030886 13087 webserver.cc:460] Webserver started at http://127.28.25.1:36543/ using document root <none> and password file <none>
I20260811 11:18:40.031279 13087 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:40.031415 13087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:40.033800 13087 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:18:40.034906 13136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:40.035202 13087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:40.035313 13087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "e93aad5aa4844bc08cb3d5d64a4b262f"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "ec44e00ff57330148b8c8b9331f3804d"
server_key_iv: "15276159fde0a2117433fcadfc65538e"
server_key_version: "encryptionkey@0"
I20260811 11:18:40.035863 13087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:40.081707 13087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:40.084791 13087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:40.085004 13087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:40.085300 13087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:40.085851 13145 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 11:18:40.088171 13087 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 11:18:40.088253 13087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260811 11:18:40.088301 13087 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 11:18:40.089059 13087 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 11:18:40.089120 13087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260811 11:18:40.089200 13145 tablet_bootstrap.cc:492] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Bootstrap starting.
I20260811 11:18:40.091277 13145 log.cc:824] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:40.094923 13145 tablet_bootstrap.cc:492] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:18:40.095432 13145 tablet_bootstrap.cc:492] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Bootstrap complete.
I20260811 11:18:40.095625 13145 ts_tablet_manager.cc:1397] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260811 11:18:40.097024 13087 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:36293
I20260811 11:18:40.097389 13087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:18:40.097817 13262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:36293 every 8 connection(s)
I20260811 11:18:40.098001 13145 raft_consensus.cc:348] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:40.098429 13145 raft_consensus.cc:729] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93aad5aa4844bc08cb3d5d64a4b262f, State: Initialized, Role: FOLLOWER
I20260811 11:18:40.098580 13145 consensus_queue.cc:260] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:40.098858 13145 ts_tablet_manager.cc:1428] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260811 11:18:40.107108 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 13087
I20260811 11:18:40.107467 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:43589
--local_ip_for_outbound_sockets=127.28.25.2
--tserver_master_addrs=127.28.25.62:39205
--webserver_port=34297
--webserver_interface=127.28.25.2
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:40.199489 13269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:40.199719 13269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:40.199747 13269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:40.201049 13269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:40.201099 13269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:40.201128 13269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
W20260811 11:18:40.202355 13266 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:18:40.098160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:49769)
0811 11:18:40.098360 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:40.098396 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:40.098577 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:40.202129 (+103552us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":113,"threads_started":1}
W20260811 11:18:40.202659 13263 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:39205 (0 consecutive failures): Network error: Failed to ping master at 127.28.25.62:39205: Client connection negotiation failed: client connection to 127.28.25.62:39205: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 11:18:40.203150 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:40.203019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:53399)
0811 11:18:40.203064 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:40.203120 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260811 11:18:40.203178 13269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:43589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=34297
--tserver_master_addrs=127.28.25.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:40.203451 13269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:40.203611 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:40.203479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:53945)
0811 11:18:40.203557 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:40.203580 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260811 11:18:40.203743 13269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:40.203732 13263 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:18:40.209955 13274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:40.209930 13275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:40.209916 13269 server_base.cc:1034] running on GCE node
W20260811 11:18:40.209896 13277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:40.210438 13269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:40.210677 13269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:40.211823 13269 hybrid_clock.cc:648] HybridClock initialized: now 1786447120211808 us; error 42 us; skew 500 ppm
Aug 11 11:18:40 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447120, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:40.214583 13269 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:18:40.215195 13269 webserver.cc:460] Webserver started at http://127.28.25.2:34297/ using document root <none> and password file <none>
I20260811 11:18:40.215453 13269 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:40.215503 13269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:40.216956 13269 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:40.217671 13286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:40.217932 13269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:40.218021 13269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "e9ffae368ba74ac8b9245c80111586e3"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "f0c0e85aae0d4a681a26a7307ac6f880"
server_key_iv: "ce067474f6650d10cf9467d353443a06"
server_key_version: "encryptionkey@0"
I20260811 11:18:40.218389 13269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:40.247299 13269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:40.249811 13269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:40.249948 13269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:40.250159 13269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:40.250505 13269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:40.250541 13269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:40.250602 13269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:40.250624 13269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:40.257004 13269 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:43589
I20260811 11:18:40.257128 13409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:43589 every 8 connection(s)
I20260811 11:18:40.257470 13269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:18:40.258652 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:40.258225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:54395)
0811 11:18:40.258404 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:40.258502 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
W20260811 11:18:40.259153 13410 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:39205 (0 consecutive failures): Network error: Failed to ping master at 127.28.25.62:39205: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
I20260811 11:18:40.259464 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:40.259354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:60087)
0811 11:18:40.259405 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:40.259432 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260811 11:18:40.259743 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:40.259661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:47879)
0811 11:18:40.259703 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:40.259725 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260811 11:18:40.259857 13410 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 11:18:40.265123 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 13269
I20260811 11:18:40.265375 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:44201
--local_ip_for_outbound_sockets=127.28.25.3
--tserver_master_addrs=127.28.25.62:39205
--webserver_port=44765
--webserver_interface=127.28.25.3
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:40.351311 13414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:40.351526 13414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:40.351553 13414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:40.352800 13414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:40.352852 13414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:40.352880 13414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:18:40.354434 13414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:44201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=44765
--tserver_master_addrs=127.28.25.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:40.354777 13414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:40.355096 13414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:40.360594 13421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:40.360625 13423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:40.360642 13414 server_base.cc:1034] running on GCE node
W20260811 11:18:40.360721 13420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:40.361188 13414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:40.361478 13414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:40.362658 13414 hybrid_clock.cc:648] HybridClock initialized: now 1786447120362658 us; error 38 us; skew 500 ppm
Aug 11 11:18:40 dist-test-slave-2w3w krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447120, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:40.365563 13414 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:18:40.366060 13414 webserver.cc:460] Webserver started at http://127.28.25.3:44765/ using document root <none> and password file <none>
I20260811 11:18:40.366355 13414 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:40.366437 13414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:40.367942 13414 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:18:40.368602 13432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:40.368782 13414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:18:40.368887 13414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "218c9e30682044c3bab931618da129cb"
format_stamp: "Formatted at 2026-08-11 11:18:38 on dist-test-slave-2w3w"
server_key: "b9764fa3f7311338083796b2bde883dc"
server_key_iv: "8941f156f64e94ef96a15ffe1555c074"
server_key_version: "encryptionkey@0"
I20260811 11:18:40.369251 13414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:40.388381 13414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:40.391062 13414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:40.391243 13414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:40.391513 13414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:40.391846 13414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:40.391902 13414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:40.391940 13414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:40.391985 13414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:40.398351 13414 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:44201
I20260811 11:18:40.398475 13552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:44201 every 8 connection(s)
I20260811 11:18:40.398818 13414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:18:40.400027 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:40.399613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:51331)
0811 11:18:40.399788 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:40.399876 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
W20260811 11:18:40.400507 13553 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:39205 (0 consecutive failures): Network error: Failed to ping master at 127.28.25.62:39205: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
I20260811 11:18:40.400853 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:40.400756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:55057)
0811 11:18:40.400807 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:40.400834 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 11:18:40.401119 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:40.401071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:37127)
0811 11:18:40.401090 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:40.401108 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260811 11:18:40.401306 13553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 11:18:40.402019 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 13414
I20260811 11:18:41.204414 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:41.204142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:53875)
0811 11:18:41.204295 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:41.204345 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260811 11:18:41.260520 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:41.260247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:48143)
0811 11:18:41.260407 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:41.260456 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260811 11:18:41.401981 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:41.401741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:37167)
0811 11:18:41.401872 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:41.401910 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260811 11:18:41.663559 13561 raft_consensus.cc:482] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:41.663692 13561 raft_consensus.cc:504] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:41.664105 13561 leader_election.cc:288] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218c9e30682044c3bab931618da129cb (127.28.25.3:44201), e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589)
Aug 11 11:18:41 dist-test-slave-2w3w krb5kdc[11809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447120, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.3@KRBTEST.COM
Aug 11 11:18:41 dist-test-slave-2w3w krb5kdc[11809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447120, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.2@KRBTEST.COM
I20260811 11:18:41.677098 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:41.664550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:43375 (local address 127.28.25.2:43589)
0811 11:18:41.664694 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:41.664702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:41.664729 (+ 27us) server_negotiation.cc:407] Connection header received
0811 11:18:41.665216 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:41.665218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:41.665267 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:41.665425 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:41.666346 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:41.667308 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:41.668356 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:41.668900 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:41.669029 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:41.672082 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:41.672089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:41.672089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:41.672104 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:41.674585 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:41.675382 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:41.675387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:41.675388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:41.675524 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:41.675900 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:41.675902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:41.675902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:41.676268 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:41.676368 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:41.676817 (+ 449us) server_negotiation.cc:299] Negotiation successful
0811 11:18:41.676978 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260811 11:18:41.677105 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:41.664464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:37267 (local address 127.28.25.3:44201)
0811 11:18:41.664641 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:41.664653 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:41.664694 (+ 41us) server_negotiation.cc:407] Connection header received
0811 11:18:41.665165 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:41.665167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:41.665212 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:41.665382 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:41.666156 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:41.667136 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:41.668358 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:41.668934 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:41.669067 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:41.672092 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:41.672095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:41.672096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:41.672107 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:41.674567 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:41.675413 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:41.675416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:41.675417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:41.675523 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:41.675900 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:41.675902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:41.675902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:41.676268 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:41.676368 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:41.676817 (+ 449us) server_negotiation.cc:299] Negotiation successful
0811 11:18:41.676978 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":59,"threads_started":1}
I20260811 11:18:41.677090 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:41.664328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:44201 (local address 127.28.25.1:37267)
0811 11:18:41.664509 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:41.664520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:41.664586 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:41.665438 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:41.665441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:41.665789 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:41.666010 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:41.666017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:41.667266 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:41.667267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:41.668259 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:41.668261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:41.669022 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:41.669023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:41.669138 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:41.669833 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:41.669846 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:41.671628 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:41.674689 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:41.674703 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:41.674706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:41.675280 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:41.675610 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:41.675611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:41.675611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:41.675795 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:41.676359 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:41.676369 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:41.676571 (+ 202us) client_negotiation.cc:769] Sending connection context
0811 11:18:41.676709 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 11:18:41.676948 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":86,"thread_start_us":93,"threads_started":1}
I20260811 11:18:41.677106 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:41.664328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:43589 (local address 127.28.25.1:43375)
0811 11:18:41.664433 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:41.664449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:41.664532 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:41.665438 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:41.665441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:41.666146 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:41.666210 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:41.666213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:41.667450 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:41.667451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:41.668259 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:41.668261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:41.668999 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:41.669000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:41.669137 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:41.669875 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:41.669881 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:41.671959 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:41.674689 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:41.674703 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:41.674706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:41.675280 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:41.675610 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:41.675611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:41.675611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:41.675795 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:41.676359 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:41.676369 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:41.676571 (+ 202us) client_negotiation.cc:769] Sending connection context
0811 11:18:41.676709 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 11:18:41.677007 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":122}
W20260811 11:18:41.678491 13138 leader_election.cc:341] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 218c9e30682044c3bab931618da129cb (127.28.25.3:44201): Not found: Tablet not found: 388dfddea44743eca52461ed3acd1785
W20260811 11:18:41.678656 13142 leader_election.cc:341] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589): Not found: Tablet not found: 388dfddea44743eca52461ed3acd1785
I20260811 11:18:41.678732 13142 leader_election.cc:302] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e93aad5aa4844bc08cb3d5d64a4b262f; no voters: 218c9e30682044c3bab931618da129cb, e9ffae368ba74ac8b9245c80111586e3
I20260811 11:18:41.678857 13561 raft_consensus.cc:2726] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:18:42.205204 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:42.204978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:42487)
0811 11:18:42.205111 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:42.205154 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260811 11:18:42.261356 13567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:42.261092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:45341)
0811 11:18:42.261250 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:42.261310 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260811 11:18:42.402834 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:42.402538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:40917)
0811 11:18:42.402712 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:42.402767 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260811 11:18:43.206132 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:43.205866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:46147)
0811 11:18:43.206033 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:43.206081 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260811 11:18:43.262166 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:43.261920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:35161)
0811 11:18:43.262069 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:43.262116 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260811 11:18:43.403620 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:43.403362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:34063)
0811 11:18:43.403522 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:43.403570 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260811 11:18:43.567447 13573 raft_consensus.cc:482] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:43.567564 13573 raft_consensus.cc:504] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:43.567755 13573 leader_election.cc:288] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218c9e30682044c3bab931618da129cb (127.28.25.3:44201), e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589)
W20260811 11:18:43.568244 13138 leader_election.cc:341] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 218c9e30682044c3bab931618da129cb (127.28.25.3:44201): Not found: Tablet not found: 388dfddea44743eca52461ed3acd1785
W20260811 11:18:43.568349 13142 leader_election.cc:341] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589): Not found: Tablet not found: 388dfddea44743eca52461ed3acd1785
I20260811 11:18:43.568427 13142 leader_election.cc:302] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e93aad5aa4844bc08cb3d5d64a4b262f; no voters: 218c9e30682044c3bab931618da129cb, e9ffae368ba74ac8b9245c80111586e3
I20260811 11:18:43.568535 13573 raft_consensus.cc:2726] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:18:44.207252 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:44.206941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:44787)
0811 11:18:44.207155 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:44.207201 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":143,"threads_started":1}
I20260811 11:18:44.263078 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:44.262800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:50655)
0811 11:18:44.262980 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:44.263030 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260811 11:18:44.404434 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:44.404143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:59575)
0811 11:18:44.404326 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:44.404383 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260811 11:18:45.208350 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:45.208003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:57451)
0811 11:18:45.208198 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:45.208298 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260811 11:18:45.263929 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:45.263637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:53535)
0811 11:18:45.263829 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:45.263878 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
I20260811 11:18:45.405366 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:45.405088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:56567)
0811 11:18:45.405264 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:45.405312 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260811 11:18:45.755421 13581 raft_consensus.cc:482] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:45.755534 13581 raft_consensus.cc:504] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:45.755725 13581 leader_election.cc:288] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218c9e30682044c3bab931618da129cb (127.28.25.3:44201), e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589)
W20260811 11:18:45.756141 13138 leader_election.cc:341] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 218c9e30682044c3bab931618da129cb (127.28.25.3:44201): Not found: Tablet not found: 388dfddea44743eca52461ed3acd1785
W20260811 11:18:45.756235 13142 leader_election.cc:341] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589): Not found: Tablet not found: 388dfddea44743eca52461ed3acd1785
I20260811 11:18:45.756319 13142 leader_election.cc:302] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e93aad5aa4844bc08cb3d5d64a4b262f; no voters: 218c9e30682044c3bab931618da129cb, e9ffae368ba74ac8b9245c80111586e3
I20260811 11:18:45.756408 13581 raft_consensus.cc:2726] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:18:46.209255 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:46.208976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:51665)
0811 11:18:46.209156 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:46.209205 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
I20260811 11:18:46.264914 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:46.264625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:33267)
0811 11:18:46.264800 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:46.264852 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260811 11:18:46.406221 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:46.405979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:58611)
0811 11:18:46.406128 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:46.406172 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260811 11:18:47.210354 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:47.210000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:32783)
0811 11:18:47.210214 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:47.210269 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
I20260811 11:18:47.265825 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:47.265544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:34007)
0811 11:18:47.265707 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:47.265769 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260811 11:18:47.407435 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:47.407128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:47555)
0811 11:18:47.407333 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:47.407384 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
I20260811 11:18:47.608703 13589 raft_consensus.cc:482] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:47.608821 13589 raft_consensus.cc:504] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:47.609040 13589 leader_election.cc:288] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218c9e30682044c3bab931618da129cb (127.28.25.3:44201), e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589)
W20260811 11:18:47.609521 13138 leader_election.cc:341] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 218c9e30682044c3bab931618da129cb (127.28.25.3:44201): Not found: Tablet not found: 388dfddea44743eca52461ed3acd1785
W20260811 11:18:47.609613 13142 leader_election.cc:341] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589): Not found: Tablet not found: 388dfddea44743eca52461ed3acd1785
I20260811 11:18:47.609700 13142 leader_election.cc:302] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e93aad5aa4844bc08cb3d5d64a4b262f; no voters: 218c9e30682044c3bab931618da129cb, e9ffae368ba74ac8b9245c80111586e3
I20260811 11:18:47.609781 13589 raft_consensus.cc:2726] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:18:48.211408 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:48.211130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:45587)
0811 11:18:48.211295 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:48.211354 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260811 11:18:48.266814 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:48.266552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:60905)
0811 11:18:48.266711 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:48.266762 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260811 11:18:48.408324 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:48.408072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:53247)
0811 11:18:48.408232 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:48.408274 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20260811 11:18:49.212384 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:49.212121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:45573)
0811 11:18:49.212268 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:49.212331 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260811 11:18:49.267701 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:49.267447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:44221)
0811 11:18:49.267601 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:49.267651 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260811 11:18:49.409286 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:49.408995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:37143)
0811 11:18:49.409181 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:49.409232 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":103,"threads_started":1}
I20260811 11:18:50.213265 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:50.213016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:33475)
0811 11:18:50.213168 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:50.213212 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260811 11:18:50.268678 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:50.268420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:54845)
0811 11:18:50.268573 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:50.268624 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260811 11:18:50.410123 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:50.409880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:57849)
0811 11:18:50.410027 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:50.410075 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260811 11:18:51.214176 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:51.213907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:59625)
0811 11:18:51.214066 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:51.214123 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260811 11:18:51.269573 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:51.269307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:34123)
0811 11:18:51.269480 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:51.269523 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
I20260811 11:18:51.411067 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:51.410720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:32813)
0811 11:18:51.410899 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:51.410939 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260811 11:18:52.215132 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:52.214763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:51309)
0811 11:18:52.214948 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:52.215078 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260811 11:18:52.270517 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:52.270218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:40969)
0811 11:18:52.270414 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:52.270463 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":98,"threads_started":1}
I20260811 11:18:52.411964 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:52.411632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:40799)
0811 11:18:52.411862 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:52.411913 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
I20260811 11:18:53.216195 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:53.215888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:46391)
0811 11:18:53.216093 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:53.216135 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260811 11:18:53.271541 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:53.271198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:56245)
0811 11:18:53.271409 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:53.271452 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":115,"threads_started":1}
I20260811 11:18:53.412865 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:53.412610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:41569)
0811 11:18:53.412769 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:53.412817 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260811 11:18:54.217283 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:54.216850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:47819)
0811 11:18:54.217189 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:54.217231 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":74,"threads_started":1}
I20260811 11:18:54.272528 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:54.272222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:33737)
0811 11:18:54.272425 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:54.272477 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
I20260811 11:18:54.413833 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:54.413507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:56801)
0811 11:18:54.413710 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:54.413774 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260811 11:18:54.418026 13614 raft_consensus.cc:482] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:54.418167 13614 raft_consensus.cc:504] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218c9e30682044c3bab931618da129cb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 44201 } } peers { permanent_uuid: "e9ffae368ba74ac8b9245c80111586e3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43589 } } peers { permanent_uuid: "e93aad5aa4844bc08cb3d5d64a4b262f" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36293 } }
I20260811 11:18:54.418448 13614 leader_election.cc:288] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218c9e30682044c3bab931618da129cb (127.28.25.3:44201), e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589)
W20260811 11:18:54.419051 13138 leader_election.cc:341] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 218c9e30682044c3bab931618da129cb (127.28.25.3:44201): Not found: Tablet not found: 388dfddea44743eca52461ed3acd1785
W20260811 11:18:54.419158 13142 leader_election.cc:341] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e9ffae368ba74ac8b9245c80111586e3 (127.28.25.2:43589): Not found: Tablet not found: 388dfddea44743eca52461ed3acd1785
I20260811 11:18:54.419258 13142 leader_election.cc:302] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e93aad5aa4844bc08cb3d5d64a4b262f; no voters: 218c9e30682044c3bab931618da129cb, e9ffae368ba74ac8b9245c80111586e3
I20260811 11:18:54.419368 13614 raft_consensus.cc:2726] T 388dfddea44743eca52461ed3acd1785 P e93aad5aa4844bc08cb3d5d64a4b262f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:18:55.218242 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:55.218003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.1:36497)
0811 11:18:55.218147 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:55.218194 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260811 11:18:55.273568 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:55.273286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.2:36571)
0811 11:18:55.273460 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:55.273512 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260811 11:18:55.409051 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 13087
I20260811 11:18:55.414932 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:55.414616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:39205 (local address 127.28.25.3:49537)
0811 11:18:55.414822 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:55.414881 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":115,"threads_started":1}
I20260811 11:18:55.417695 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 13269
I20260811 11:18:55.424901 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 13414
2026-08-11T11:18:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18648 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:18:55 dist-test-slave-2w3w krb5kdc[13621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:18:55 dist-test-slave-2w3w krb5kdc[13621](info): set up 2 sockets
Aug 11 11:18:55 dist-test-slave-2w3w krb5kdc[13621](info): commencing operation
krb5kdc: starting...
W20260811 11:18:56.574851 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:18:56 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:18:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:18:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:18:56.723204 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33593
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:33593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:56.811295 13634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:56.811587 13634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:56.811633 13634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:56.812960 13634 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 11:18:56.813048 13634 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 11:18:56.813071 13634 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 11:18:56.813090 13634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:18:56.813119 13634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:56.813179 13634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:56.813226 13634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:18:56.813261 13634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:18:56.815055 13634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:33593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:56.815398 13634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:56.815667 13634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:18:56.821697 13634 server_base.cc:1034] running on GCE node
W20260811 11:18:56.821646 13640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:56.821647 13642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:56.821653 13639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:56.822288 13634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:56.822638 13634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:56.823915 13634 hybrid_clock.cc:648] HybridClock initialized: now 1786447136823883 us; error 51 us; skew 500 ppm
Aug 11 11:18:56 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447136, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:56.827054 13634 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:18:56.827562 13634 webserver.cc:460] Webserver started at http://127.28.25.62:46553/ using document root <none> and password file <none>
I20260811 11:18:56.827881 13634 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:56.827939 13634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:56.828047 13634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:56.829273 13634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd"
format_stamp: "Formatted at 2026-08-11 11:18:56 on dist-test-slave-2w3w"
server_key: "bc8ea00b4f5d1594f9bec30b61a8c9e5"
server_key_iv: "6b7652cde7038f004bc2e10753fe4843"
server_key_version: "encryptionkey@0"
I20260811 11:18:56.829628 13634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd"
format_stamp: "Formatted at 2026-08-11 11:18:56 on dist-test-slave-2w3w"
server_key: "bc8ea00b4f5d1594f9bec30b61a8c9e5"
server_key_iv: "6b7652cde7038f004bc2e10753fe4843"
server_key_version: "encryptionkey@0"
I20260811 11:18:56.831369 13634 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:18:56.832275 13649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:56.832497 13634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:56.832595 13634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd"
format_stamp: "Formatted at 2026-08-11 11:18:56 on dist-test-slave-2w3w"
server_key: "bc8ea00b4f5d1594f9bec30b61a8c9e5"
server_key_iv: "6b7652cde7038f004bc2e10753fe4843"
server_key_version: "encryptionkey@0"
I20260811 11:18:56.832664 13634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:56.856988 13634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:56.859679 13634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:56.859814 13634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:56.864403 13634 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:33593
I20260811 11:18:56.864470 13709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:33593 every 8 connection(s)
I20260811 11:18:56.864858 13634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:18:56.865892 13710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:56.868404 13710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Bootstrap starting.
I20260811 11:18:56.869072 13710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:56.869402 13710 log.cc:824] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:56.870198 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 13634
I20260811 11:18:56.870303 13710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: No bootstrap required, opened a new log
I20260811 11:18:56.870319 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:18:56.870540 28772 external_mini_cluster.cc:1428] Setting key 96a48a2165773fbed394e9214b82e3cf
I20260811 11:18:56.873234 13710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:18:56.873394 13710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:56.873418 13710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f10b1ad86643dc8e65abe9a6f9e7dd, State: Initialized, Role: FOLLOWER
I20260811 11:18:56.873557 13710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:18:56.873638 13710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:18:56.873684 13710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:18:56.873735 13710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:56.874447 13710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:18:56.874552 13710 leader_election.cc:302] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f10b1ad86643dc8e65abe9a6f9e7dd; no voters:
I20260811 11:18:56.874894 13710 leader_election.cc:288] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:18:56.874925 13715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:18:56.875070 13715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 1 LEADER]: Becoming Leader. State: Replica: 74f10b1ad86643dc8e65abe9a6f9e7dd, State: Running, Role: LEADER
I20260811 11:18:56.875227 13715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:18:56.875275 13710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:18:56.875895 13716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } } }
I20260811 11:18:56.876207 13716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:56.876120 13717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f10b1ad86643dc8e65abe9a6f9e7dd. Latest consensus state: current_term: 1 leader_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } } }
I20260811 11:18:56.876541 13717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: This master's current role is: LEADER
I20260811 11:18:56.876699 13726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:18:56.877332 13726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 11 11:18:56 dist-test-slave-2w3w krb5kdc[13621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:56.879325 13726 catalog_manager.cc:1347] Generated new cluster ID: 489d82000381492fae9cb430f574f011
I20260811 11:18:56.879379 13726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:18:56.883708 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:56.871215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.28.25.62:33593)
0811 11:18:56.871451 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:56.871460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:56.871489 (+ 29us) server_negotiation.cc:407] Connection header received
0811 11:18:56.871977 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:56.871984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:56.872043 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:56.872167 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:56.873083 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:56.874536 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:56.875623 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:56.876328 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:56.876520 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:56.879816 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:56.879820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:56.879821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:56.879834 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:56.881895 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:56.882369 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:56.882372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:56.882373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:56.882479 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:56.882712 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:56.882713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:56.882713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:56.883057 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:56.883192 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:56.883334 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 11:18:56.883480 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":103,"threads_started":1}
I20260811 11:18:56.885900 13726 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:18:56.886422 13726 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:18:56.889581 13726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Generated new TSK 0
I20260811 11:18:56.889787 13726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:18:56.931543 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33593
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:57.030385 13737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:57.030676 13737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:57.030725 13737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:57.032481 13737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:57.032590 13737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:57.032645 13737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:18:57.034476 13737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:57.034821 13737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:57.035171 13737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:57.041231 13744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:57.041316 13737 server_base.cc:1034] running on GCE node
W20260811 11:18:57.041304 13746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:57.041239 13742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:57.041711 13737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:57.041960 13737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:57.043182 13737 hybrid_clock.cc:648] HybridClock initialized: now 1786447137043169 us; error 33 us; skew 500 ppm
Aug 11 11:18:57 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:57.047091 13737 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:18:57.047787 13737 webserver.cc:460] Webserver started at http://127.28.25.1:43283/ using document root <none> and password file <none>
I20260811 11:18:57.048134 13737 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:57.048233 13737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:57.048415 13737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:57.050109 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "efbc2cdc45804ab3b5ba931af9f808f7"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "9b32f3c9a6ae4cc017b85eac7bdbc65c"
server_key_iv: "bc128e0eb26d5cff8f3c61d062664d86"
server_key_version: "encryptionkey@0"
I20260811 11:18:57.050575 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "efbc2cdc45804ab3b5ba931af9f808f7"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "9b32f3c9a6ae4cc017b85eac7bdbc65c"
server_key_iv: "bc128e0eb26d5cff8f3c61d062664d86"
server_key_version: "encryptionkey@0"
I20260811 11:18:57.052749 13737 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 11:18:57.053903 13753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:57.054085 13737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:57.054191 13737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "efbc2cdc45804ab3b5ba931af9f808f7"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "9b32f3c9a6ae4cc017b85eac7bdbc65c"
server_key_iv: "bc128e0eb26d5cff8f3c61d062664d86"
server_key_version: "encryptionkey@0"
I20260811 11:18:57.054286 13737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:57.071172 13737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:57.073755 13737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:57.073949 13737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:57.074266 13737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:57.074622 13737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:57.074681 13737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:57.074739 13737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:57.074767 13737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:57.081470 13737 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:44075
I20260811 11:18:57.081540 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:44075 every 8 connection(s)
I20260811 11:18:57.081938 13737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
Aug 11 11:18:57 dist-test-slave-2w3w krb5kdc[13621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:57.089658 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 13737
I20260811 11:18:57.089876 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:18:57.090101 28772 external_mini_cluster.cc:1428] Setting key b118d9e38c8466ea3d92748651f1ec76
I20260811 11:18:57.094564 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.083030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:59531 (local address 127.28.25.62:33593)
0811 11:18:57.083100 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:57.083103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:57.083337 (+ 234us) server_negotiation.cc:407] Connection header received
0811 11:18:57.083926 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:57.083928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:57.083940 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:57.084016 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:57.085199 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.085736 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.087041 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.087306 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.087381 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.090474 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:57.090476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:57.090476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:57.090487 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:57.092514 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:57.093297 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:57.093299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:57.093299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:57.093361 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:57.093716 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:57.093717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:57.093717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:57.093836 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:57.093889 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:57.094333 (+ 444us) server_negotiation.cc:299] Negotiation successful
0811 11:18:57.094454 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 11:18:57.095062 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.082939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:59531)
0811 11:18:57.083189 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:57.083233 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:57.083792 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:57.084111 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:57.084115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:57.084792 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:57.085088 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.085094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.085842 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.085844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.086832 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.086835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.087430 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.087431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.087583 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.088231 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:57.088241 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:57.090291 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:57.092669 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:57.092681 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:57.092683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:57.093189 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:57.093453 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:57.093454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:57.093455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:57.093641 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:57.093914 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:57.093922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:57.094141 (+ 219us) client_negotiation.cc:769] Sending connection context
0811 11:18:57.094292 (+ 151us) client_negotiation.cc:240] Negotiation successful
0811 11:18:57.094511 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":118,"threads_started":1}
I20260811 11:18:57.095721 13878 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33593
I20260811 11:18:57.095853 13878 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:57.096095 13878 heartbeater.cc:511] Master 127.28.25.62:33593 requested a full tablet report, sending...
I20260811 11:18:57.096870 13670 ts_manager.cc:194] Registered new tserver with Master: efbc2cdc45804ab3b5ba931af9f808f7 (127.28.25.1:44075)
I20260811 11:18:57.097901 13670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:59531
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:18:57.123602 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33593
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:57.212584 13885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:57.212827 13885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:57.212855 13885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:57.214100 13885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:57.214210 13885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:57.214260 13885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:18:57.215970 13885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:57.216244 13885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:57.216488 13885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:57.222211 13891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:57.222260 13892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:57.222280 13885 server_base.cc:1034] running on GCE node
W20260811 11:18:57.222211 13894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:57.222733 13885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:57.223089 13885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:57.224287 13885 hybrid_clock.cc:648] HybridClock initialized: now 1786447137224273 us; error 33 us; skew 500 ppm
Aug 11 11:18:57 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:57.227159 13885 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:18:57.227679 13885 webserver.cc:460] Webserver started at http://127.28.25.2:38377/ using document root <none> and password file <none>
I20260811 11:18:57.227932 13885 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:57.228006 13885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:57.228165 13885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:57.229326 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "deefe38bc52947db99da0880d75201f7"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "11128e479836bdcffde7ea233a49c7f9"
server_key_iv: "cd4815d63c63941536fcff6d0bfe24b5"
server_key_version: "encryptionkey@0"
I20260811 11:18:57.229728 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "deefe38bc52947db99da0880d75201f7"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "11128e479836bdcffde7ea233a49c7f9"
server_key_iv: "cd4815d63c63941536fcff6d0bfe24b5"
server_key_version: "encryptionkey@0"
I20260811 11:18:57.231418 13885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260811 11:18:57.232472 13901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:57.232808 13885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:57.232944 13885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "deefe38bc52947db99da0880d75201f7"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "11128e479836bdcffde7ea233a49c7f9"
server_key_iv: "cd4815d63c63941536fcff6d0bfe24b5"
server_key_version: "encryptionkey@0"
I20260811 11:18:57.233039 13885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:57.259717 13885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:57.262365 13885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:57.262576 13885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:57.262894 13885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:57.263316 13885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:57.263379 13885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:57.263423 13885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:57.263469 13885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:57.270044 13885 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:44051
I20260811 11:18:57.270112 14022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:44051 every 8 connection(s)
I20260811 11:18:57.270546 13885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:18:57.270807 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 13885
I20260811 11:18:57.270931 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:18:57.271202 28772 external_mini_cluster.cc:1428] Setting key 3b38a46db21c97e5d7cdc0091063edd3
Aug 11 11:18:57 dist-test-slave-2w3w krb5kdc[13621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
I20260811 11:18:57.284173 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.271511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:60667 (local address 127.28.25.62:33593)
0811 11:18:57.271629 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:57.271632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:57.271998 (+ 366us) server_negotiation.cc:407] Connection header received
0811 11:18:57.273008 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:57.273010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:57.273026 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:57.273111 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:57.275073 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.275954 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.277417 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.277721 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.277808 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.280699 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:57.280701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:57.280701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:57.280711 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:57.282411 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:57.282923 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:57.282924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:57.282925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:57.283068 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:57.283388 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:57.283389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:57.283390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:57.283540 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:57.283629 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:57.283961 (+ 332us) server_negotiation.cc:299] Negotiation successful
0811 11:18:57.284064 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260811 11:18:57.284420 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.271541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:60667)
0811 11:18:57.271804 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:57.271901 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:57.272837 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:57.273276 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:57.273281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:57.274479 (+ 1198us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:57.274848 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.274857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.276086 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.276087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.277303 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.277307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.277838 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.277839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.278012 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.278663 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:57.278671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:57.280568 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:57.282526 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:57.282538 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:57.282541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:57.282829 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:57.283161 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:57.283162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:57.283163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:57.283308 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:57.283619 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:57.283628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:57.283804 (+ 176us) client_negotiation.cc:769] Sending connection context
0811 11:18:57.283924 (+ 120us) client_negotiation.cc:240] Negotiation successful
0811 11:18:57.284135 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
I20260811 11:18:57.285071 14023 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33593
I20260811 11:18:57.285173 14023 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:57.285423 14023 heartbeater.cc:511] Master 127.28.25.62:33593 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
I20260811 11:18:57.285970 13670 ts_manager.cc:194] Registered new tserver with Master: deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051)
I20260811 11:18:57.286369 13670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:60667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:18:57.305789 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33593
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:18:57.394680 14028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:18:57.394934 14028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:18:57.395021 14028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:18:57.396221 14028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:18:57.396270 14028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:18:57.396299 14028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:18:57.397965 14028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:18:57.398216 14028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:18:57.398416 14028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:18:57.404033 14035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:57.404134 14028 server_base.cc:1034] running on GCE node
W20260811 11:18:57.404042 14034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:18:57.404018 14038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:18:57.404506 14028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:18:57.404830 14028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:18:57.405995 14028 hybrid_clock.cc:648] HybridClock initialized: now 1786447137405976 us; error 30 us; skew 500 ppm
Aug 11 11:18:57 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:18:57.409066 14028 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:18:57.409596 14028 webserver.cc:460] Webserver started at http://127.28.25.3:36649/ using document root <none> and password file <none>
I20260811 11:18:57.409864 14028 fs_manager.cc:359] Metadata directory not provided
I20260811 11:18:57.409938 14028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:18:57.410108 14028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:18:57.411371 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "b1a9983e15044ca098da92be7c1e0592"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "6b0aa9f6b16cc638113bc9ea4e6a59d4"
server_key_iv: "7a335d0638ff06ab876c393447d22a61"
server_key_version: "encryptionkey@0"
I20260811 11:18:57.411763 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "b1a9983e15044ca098da92be7c1e0592"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "6b0aa9f6b16cc638113bc9ea4e6a59d4"
server_key_iv: "7a335d0638ff06ab876c393447d22a61"
server_key_version: "encryptionkey@0"
I20260811 11:18:57.413326 14028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:18:57.414324 14045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:57.414559 14028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:18:57.414683 14028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "b1a9983e15044ca098da92be7c1e0592"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "6b0aa9f6b16cc638113bc9ea4e6a59d4"
server_key_iv: "7a335d0638ff06ab876c393447d22a61"
server_key_version: "encryptionkey@0"
I20260811 11:18:57.414790 14028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:18:57.433198 14028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:18:57.435911 14028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:18:57.436205 14028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:18:57.436512 14028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:18:57.436878 14028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:18:57.436913 14028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:57.436937 14028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:18:57.436985 14028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:18:57.443738 14028 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:46095
I20260811 11:18:57.443804 14166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:46095 every 8 connection(s)
I20260811 11:18:57.444160 14028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
Aug 11 11:18:57 dist-test-slave-2w3w krb5kdc[13621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:18:57.452682 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 14028
I20260811 11:18:57.452816 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:18:57.453115 28772 external_mini_cluster.cc:1428] Setting key 412083dc9b46ec123b11e3c0644073fe
I20260811 11:18:57.455235 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.445044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:37345 (local address 127.28.25.62:33593)
0811 11:18:57.445110 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:57.445113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:57.445355 (+ 242us) server_negotiation.cc:407] Connection header received
0811 11:18:57.445904 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:57.445905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:57.445918 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:57.445990 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:57.447117 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.447696 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.448918 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.449131 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.449194 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.451794 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:57.451796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:57.451796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:57.451805 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:57.453601 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:57.454109 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:57.454110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:57.454111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:57.454245 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:57.454510 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:57.454510 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:57.454511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:57.454638 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:57.454699 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:57.455080 (+ 381us) server_negotiation.cc:299] Negotiation successful
0811 11:18:57.455161 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 11:18:57.455437 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.444935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:37345)
0811 11:18:57.445175 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:57.445221 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:57.445735 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:57.446074 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:57.446077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:57.446708 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:18:57.446927 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.446932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.447822 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.447823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.448817 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.448820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.449251 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.449252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.449367 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.449957 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:18:57.449966 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:18:57.451680 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:18:57.453674 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:57.453686 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:57.453689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:57.454020 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:57.454319 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:18:57.454320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:18:57.454320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:18:57.454444 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:18:57.454703 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:18:57.454719 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:18:57.454883 (+ 164us) client_negotiation.cc:769] Sending connection context
0811 11:18:57.455033 (+ 150us) client_negotiation.cc:240] Negotiation successful
0811 11:18:57.455189 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":97,"threads_started":1}
I20260811 11:18:57.455981 14167 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33593
I20260811 11:18:57.456099 14167 heartbeater.cc:464] Registering TS with master...
I20260811 11:18:57.456295 14167 heartbeater.cc:511] Master 127.28.25.62:33593 requested a full tablet report, sending...
I20260811 11:18:57.456758 13670 ts_manager.cc:194] Registered new tserver with Master: b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095)
I20260811 11:18:57.457283 13670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:37345
I20260811 11:18:57.465860 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:18:57.475618 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.467519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.28.25.62:33593)
0811 11:18:57.467615 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:57.467618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:57.467723 (+ 105us) server_negotiation.cc:407] Connection header received
0811 11:18:57.467829 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:57.467830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:57.467850 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:57.467932 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:18:57.468916 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.469727 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.470581 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.470813 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.470921 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.472436 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:18:57.472439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:18:57.472439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:18:57.472448 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:18:57.474094 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:57.474541 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:57.474542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:57.474543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:57.474609 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:18:57.474866 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:18:57.474867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:18:57.474868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:18:57.475131 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:18:57.475256 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:57.475397 (+ 141us) server_negotiation.cc:299] Negotiation successful
0811 11:18:57.475498 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 11:18:57.477756 13670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:18:57.479609 13670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:18:57.486770 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.482237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:44075 (local address 127.0.0.1:60324)
0811 11:18:57.482701 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:57.482711 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:57.482758 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:57.483200 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:57.483201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:57.483203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:57.483263 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.483264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.484279 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.484280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.485594 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.485597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.486488 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.486489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.486570 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.486584 (+ 14us) client_negotiation.cc:769] Sending connection context
0811 11:18:57.486651 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 11:18:57.486696 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":63,"thread_start_us":132,"threads_started":1}
I20260811 11:18:57.486871 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.482594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.28.25.1:44075)
0811 11:18:57.482913 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:57.482917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:57.482930 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:18:57.483003 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:57.483006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:57.483023 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:57.483110 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:57.483331 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.484156 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.485696 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.486397 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.486477 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.486541 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:57.486691 (+ 150us) server_negotiation.cc:299] Negotiation successful
0811 11:18:57.486739 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":118,"threads_started":1}
I20260811 11:18:57.487131 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.482236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:44051 (local address 127.0.0.1:58080)
0811 11:18:57.482446 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:57.482489 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:57.482562 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:57.482942 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:57.482945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:57.482947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:57.483051 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.483054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.484479 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.484480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.485920 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.485923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.486868 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.486869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.486938 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.486942 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:18:57.487027 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 11:18:57.487081 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260811 11:18:57.487838 13809 tablet_service.cc:1467] Processing CreateTablet for tablet adcac0b1611a4461b24504383808cd5b (DEFAULT_TABLE table=test-table [id=24b8ae657793441aa7639252bae3b69d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:57.488047 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.482451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.28.25.2:44051)
0811 11:18:57.482684 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:57.482690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:57.482703 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:18:57.482751 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:57.482753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:57.482772 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:57.482915 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:57.483150 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.484364 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.486021 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.486764 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.486842 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.486897 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:57.487910 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0811 11:18:57.487960 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":120,"threads_started":1}
I20260811 11:18:57.488080 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.482237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:46095 (local address 127.0.0.1:35692)
0811 11:18:57.482838 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:57.482847 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:57.482883 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:57.483363 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:57.483363 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:57.483365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:57.483402 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.483403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.484636 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.484636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.485923 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.485925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.487803 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.487804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.487865 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.487869 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:18:57.487955 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 11:18:57.488000 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":53,"thread_start_us":93,"threads_started":1}
I20260811 11:18:57.488273 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adcac0b1611a4461b24504383808cd5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:57.488552 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.482688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.28.25.3:46095)
0811 11:18:57.483145 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:57.483149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:57.483160 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:18:57.483200 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:57.483202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:57.483217 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:57.483367 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:57.483452 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.484524 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.486944 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.487711 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.487796 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.487854 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:57.487982 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 11:18:57.488458 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":99,"threads_started":1}
I20260811 11:18:57.488818 13953 tablet_service.cc:1467] Processing CreateTablet for tablet adcac0b1611a4461b24504383808cd5b (DEFAULT_TABLE table=test-table [id=24b8ae657793441aa7639252bae3b69d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:57.489174 13953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adcac0b1611a4461b24504383808cd5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:57.489362 14097 tablet_service.cc:1467] Processing CreateTablet for tablet adcac0b1611a4461b24504383808cd5b (DEFAULT_TABLE table=test-table [id=24b8ae657793441aa7639252bae3b69d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:18:57.489693 14097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adcac0b1611a4461b24504383808cd5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:18:57.492152 14186 tablet_bootstrap.cc:492] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Bootstrap starting.
I20260811 11:18:57.492535 14187 tablet_bootstrap.cc:492] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: Bootstrap starting.
I20260811 11:18:57.492975 14186 tablet_bootstrap.cc:654] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:57.493259 14187 tablet_bootstrap.cc:654] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:57.493621 14188 tablet_bootstrap.cc:492] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: Bootstrap starting.
I20260811 11:18:57.493727 14186 log.cc:824] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:57.493928 14187 log.cc:824] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:57.494362 14188 tablet_bootstrap.cc:654] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: Neither blocks nor log segments found. Creating new log.
I20260811 11:18:57.494796 14186 tablet_bootstrap.cc:492] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: No bootstrap required, opened a new log
I20260811 11:18:57.494923 14186 ts_tablet_manager.cc:1397] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:18:57.495079 14188 log.cc:824] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: Log is configured to *not* fsync() on all Append() calls
I20260811 11:18:57.496598 14186 raft_consensus.cc:348] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:18:57.496757 14186 raft_consensus.cc:374] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:57.496790 14186 raft_consensus.cc:729] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc2cdc45804ab3b5ba931af9f808f7, State: Initialized, Role: FOLLOWER
I20260811 11:18:57.496933 14186 consensus_queue.cc:260] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:18:57.497259 14186 ts_tablet_manager.cc:1428] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:18:57.497311 13878 heartbeater.cc:503] Master 127.28.25.62:33593 was elected leader, sending a full tablet report...
I20260811 11:18:57.497756 14187 tablet_bootstrap.cc:492] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: No bootstrap required, opened a new log
I20260811 11:18:57.497870 14187 ts_tablet_manager.cc:1397] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260811 11:18:57.499830 14188 tablet_bootstrap.cc:492] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: No bootstrap required, opened a new log
I20260811 11:18:57.499918 14188 ts_tablet_manager.cc:1397] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260811 11:18:57.500100 14187 raft_consensus.cc:348] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:18:57.500286 14187 raft_consensus.cc:374] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:57.500336 14187 raft_consensus.cc:729] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deefe38bc52947db99da0880d75201f7, State: Initialized, Role: FOLLOWER
I20260811 11:18:57.500468 14187 consensus_queue.cc:260] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:18:57.500809 14187 ts_tablet_manager.cc:1428] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:18:57.500856 14023 heartbeater.cc:503] Master 127.28.25.62:33593 was elected leader, sending a full tablet report...
I20260811 11:18:57.502283 14188 raft_consensus.cc:348] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:18:57.502460 14188 raft_consensus.cc:374] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:18:57.502486 14188 raft_consensus.cc:729] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a9983e15044ca098da92be7c1e0592, State: Initialized, Role: FOLLOWER
I20260811 11:18:57.502616 14188 consensus_queue.cc:260] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:18:57.503017 14188 ts_tablet_manager.cc:1428] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260811 11:18:57.503039 14167 heartbeater.cc:503] Master 127.28.25.62:33593 was elected leader, sending a full tablet report...
I20260811 11:18:57.518401 14194 raft_consensus.cc:482] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:18:57.518535 14194 raft_consensus.cc:504] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:18:57.519100 14194 leader_election.cc:288] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095), deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051)
W20260811 11:18:57.521500 14024 tablet.cc:2369] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:18:57.522343 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.519287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:46095 (local address 127.28.25.1:52537)
0811 11:18:57.519347 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:57.519360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:57.519406 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:57.519598 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:57.519600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:57.519602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:57.519703 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.519705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.520400 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.520402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.521250 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.521253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.522110 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.522111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.522183 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.522187 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:18:57.522238 (+ 51us) client_negotiation.cc:240] Negotiation successful
0811 11:18:57.522285 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":3}
I20260811 11:18:57.522363 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.519345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:52537 (local address 127.28.25.3:46095)
0811 11:18:57.519432 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:57.519435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:57.519448 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:18:57.519482 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:57.519483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:57.519497 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:57.519592 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:57.519788 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.520288 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.521408 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.522019 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.522101 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.522113 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:57.522264 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 11:18:57.522308 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 11:18:57.522787 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.519360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:36675 (local address 127.28.25.2:44051)
0811 11:18:57.519423 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:57.519426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:57.519588 (+ 162us) server_negotiation.cc:407] Connection header received
0811 11:18:57.519646 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:57.519648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:57.519660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:57.519717 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:18:57.519879 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.520353 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.521363 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.522019 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.522101 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.522113 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:57.522668 (+ 555us) server_negotiation.cc:299] Negotiation successful
0811 11:18:57.522718 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260811 11:18:57.522862 14119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcac0b1611a4461b24504383808cd5b" candidate_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a9983e15044ca098da92be7c1e0592" is_pre_election: true
I20260811 11:18:57.523079 14119 raft_consensus.cc:2454] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbc2cdc45804ab3b5ba931af9f808f7 in term 0.
I20260811 11:18:57.523103 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.519297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:44051 (local address 127.28.25.1:36675)
0811 11:18:57.519511 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:57.519521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:18:57.519589 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:18:57.519722 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:18:57.519722 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:18:57.519724 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:18:57.519802 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.519804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.520445 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.520446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.521260 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:18:57.521262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.522493 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.522494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:18:57.522576 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.522580 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:18:57.522932 (+ 352us) client_negotiation.cc:240] Negotiation successful
0811 11:18:57.523022 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":22,"thread_start_us":138,"threads_started":1}
I20260811 11:18:57.523356 13758 leader_election.cc:302] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1a9983e15044ca098da92be7c1e0592, efbc2cdc45804ab3b5ba931af9f808f7; no voters:
I20260811 11:18:57.523525 14194 raft_consensus.cc:2781] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:18:57.523597 14194 raft_consensus.cc:482] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:18:57.523552 13976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcac0b1611a4461b24504383808cd5b" candidate_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deefe38bc52947db99da0880d75201f7" is_pre_election: true
I20260811 11:18:57.523623 14194 raft_consensus.cc:3037] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:57.523695 13976 raft_consensus.cc:2454] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbc2cdc45804ab3b5ba931af9f808f7 in term 0.
I20260811 11:18:57.524500 14194 raft_consensus.cc:504] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:18:57.524662 14194 leader_election.cc:288] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 1 election: Requested vote from peers b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095), deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051)
I20260811 11:18:57.524849 13976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcac0b1611a4461b24504383808cd5b" candidate_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deefe38bc52947db99da0880d75201f7"
I20260811 11:18:57.524959 13976 raft_consensus.cc:3037] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:57.524961 14119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcac0b1611a4461b24504383808cd5b" candidate_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a9983e15044ca098da92be7c1e0592"
I20260811 11:18:57.525061 14119 raft_consensus.cc:3037] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:18:57.525849 13976 raft_consensus.cc:2454] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbc2cdc45804ab3b5ba931af9f808f7 in term 1.
I20260811 11:18:57.525972 14119 raft_consensus.cc:2454] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbc2cdc45804ab3b5ba931af9f808f7 in term 1.
I20260811 11:18:57.526050 13756 leader_election.cc:302] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: deefe38bc52947db99da0880d75201f7, efbc2cdc45804ab3b5ba931af9f808f7; no voters:
I20260811 11:18:57.526142 14194 raft_consensus.cc:2781] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:18:57.526312 14194 raft_consensus.cc:686] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 LEADER]: Becoming Leader. State: Replica: efbc2cdc45804ab3b5ba931af9f808f7, State: Running, Role: LEADER
I20260811 11:18:57.526396 14194 consensus_queue.cc:237] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:18:57.527453 13670 catalog_manager.cc:5495] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to efbc2cdc45804ab3b5ba931af9f808f7 (127.28.25.1). New cstate: current_term: 1 leader_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } health_report { overall_health: HEALTHY } } }
I20260811 11:18:57.539112 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.536150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.28.25.1:44075)
0811 11:18:57.536207 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 11:18:57.536209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:18:57.536225 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:18:57.536347 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:18:57.536348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:18:57.536372 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:18:57.536450 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:18:57.536712 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.537280 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.538058 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:18:57.538274 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:18:57.538379 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:18:57.538616 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:18:57.538730 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:18:57.538873 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 11:18:57.538912 (+ 39us) server_negotiation.cc:299] Negotiation successful
0811 11:18:57.538936 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 11:18:57.541354 13976 raft_consensus.cc:1261] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7 [term 1 FOLLOWER]: Refusing update from remote peer efbc2cdc45804ab3b5ba931af9f808f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:18:57.541342 14119 raft_consensus.cc:1261] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592 [term 1 FOLLOWER]: Refusing update from remote peer efbc2cdc45804ab3b5ba931af9f808f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:18:57.541775 14194 consensus_queue.cc:1035] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:18:57.541981 14198 consensus_queue.cc:1035] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:18:57.544905 14201 mvcc.cc:205] Tried to move back new op lower bound from 7317287475365990400 to 7317287475308900352. Current Snapshot: MvccSnapshot[applied={T|T < 7317287475365990400}]
I20260811 11:18:57.544921 14203 mvcc.cc:205] Tried to move back new op lower bound from 7317287475365990400 to 7317287475308900352. Current Snapshot: MvccSnapshot[applied={T|T < 7317287475365990400}]
I20260811 11:18:57.547843 13669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:18:57.547979 13669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:18:57.548888 13669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Sending DeleteTablet for 3 replicas of tablet adcac0b1611a4461b24504383808cd5b
I20260811 11:18:57.549291 14097 tablet_service.cc:1514] Processing DeleteTablet for tablet adcac0b1611a4461b24504383808cd5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:57 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:35692
I20260811 11:18:57.549427 13809 tablet_service.cc:1514] Processing DeleteTablet for tablet adcac0b1611a4461b24504383808cd5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:57 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:60324
I20260811 11:18:57.549495 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 13737
I20260811 11:18:57.549614 13953 tablet_service.cc:1514] Processing DeleteTablet for tablet adcac0b1611a4461b24504383808cd5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:57 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:58080
I20260811 11:18:57.552664 14213 tablet_replica.cc:330] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: stopping tablet replica
I20260811 11:18:57.552856 14213 raft_consensus.cc:2229] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:18:57.552996 14213 raft_consensus.cc:2258] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:18:57.553354 14212 tablet_replica.cc:330] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: stopping tablet replica
I20260811 11:18:57.553479 14212 raft_consensus.cc:2229] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:18:57.553588 14212 raft_consensus.cc:2258] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:18:57.553941 14213 ts_tablet_manager.cc:1905] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:18:57.554883 14212 ts_tablet_manager.cc:1905] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:18:57.556309 14213 ts_tablet_manager.cc:1918] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:18:57.556403 14213 log.cc:1196] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/adcac0b1611a4461b24504383808cd5b
I20260811 11:18:57.556764 14213 ts_tablet_manager.cc:1939] T adcac0b1611a4461b24504383808cd5b P deefe38bc52947db99da0880d75201f7: Deleting consensus metadata
I20260811 11:18:57.557328 13651 catalog_manager.cc:4841] TS deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051): tablet adcac0b1611a4461b24504383808cd5b (table test-table [id=24b8ae657793441aa7639252bae3b69d]) successfully deleted
I20260811 11:18:57.557608 14212 ts_tablet_manager.cc:1918] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:18:57.557693 14212 log.cc:1196] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/adcac0b1611a4461b24504383808cd5b
I20260811 11:18:57.558044 14212 ts_tablet_manager.cc:1939] T adcac0b1611a4461b24504383808cd5b P b1a9983e15044ca098da92be7c1e0592: Deleting consensus metadata
I20260811 11:18:57.558565 13654 catalog_manager.cc:4841] TS b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095): tablet adcac0b1611a4461b24504383808cd5b (table test-table [id=24b8ae657793441aa7639252bae3b69d]) successfully deleted
W20260811 11:18:57.561398 13650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:44075 (error 108)
I20260811 11:18:57.561749 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 13885
I20260811 11:18:57.563488 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:18:57.561937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:44075 (local address 127.0.0.1:60334)
0811 11:18:57.563396 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0811 11:18:57.563450 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:44075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1013}
W20260811 11:18:57.563721 13650 catalog_manager.cc:4584] TS efbc2cdc45804ab3b5ba931af9f808f7 (127.28.25.1:44075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adcac0b1611a4461b24504383808cd5b: Network error: Client connection negotiation failed: client connection to 127.28.25.1:44075: connect: Connection refused (error 111)
I20260811 11:18:57.572141 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 14028
I20260811 11:18:57.582158 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 13634
I20260811 11:19:17.592079 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33593
--webserver_interface=127.28.25.62
--webserver_port=46553
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:33593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:19:17.682001 14215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:17.682258 14215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:17.682284 14215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:17.683640 14215 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 11:19:17.683701 14215 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 11:19:17.683714 14215 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 11:19:17.683727 14215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:19:17.683740 14215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:17.683753 14215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:17.683766 14215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:19:17.683805 14215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:19:17.685518 14215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:33593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=46553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:17.685833 14215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:17.686110 14215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:19:17.699663 14215 server_base.cc:1034] running on GCE node
W20260811 11:19:17.699635 14222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:17.699635 14221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:17.699635 14224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:17.700122 14215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:17.700538 14215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:17.701725 14215 hybrid_clock.cc:648] HybridClock initialized: now 1786447157701719 us; error 46 us; skew 500 ppm
Aug 11 11:19:17 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:17.708552 14215 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:19:17.709120 14215 webserver.cc:460] Webserver started at http://127.28.25.62:46553/ using document root <none> and password file <none>
I20260811 11:19:17.709398 14215 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:17.709496 14215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:17.711175 14215 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:19:17.712148 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:17.712378 14215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:17.712497 14215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd"
format_stamp: "Formatted at 2026-08-11 11:18:56 on dist-test-slave-2w3w"
server_key: "bc8ea00b4f5d1594f9bec30b61a8c9e5"
server_key_iv: "6b7652cde7038f004bc2e10753fe4843"
server_key_version: "encryptionkey@0"
I20260811 11:19:17.712931 14215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:17.732810 14215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:17.735389 14215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:17.735579 14215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:17.740404 14215 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:33593
I20260811 11:19:17.740442 14288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:33593 every 8 connection(s)
I20260811 11:19:17.740882 14215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:19:17.742434 14289 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:19:17.743141 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Bootstrap starting.
I20260811 11:19:17.749271 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 14215
I20260811 11:19:17.749615 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:44075
--local_ip_for_outbound_sockets=127.28.25.1
--tserver_master_addrs=127.28.25.62:33593
--webserver_port=43283
--webserver_interface=127.28.25.1
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:19:17.753167 14289 log.cc:824] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:17.755566 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:19:17.755923 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Bootstrap complete.
I20260811 11:19:17.758461 14289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:19:17.758755 14289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f10b1ad86643dc8e65abe9a6f9e7dd, State: Initialized, Role: FOLLOWER
I20260811 11:19:17.758991 14289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:19:17.759115 14289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:19:17.759181 14289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:19:17.759235 14289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 1 FOLLOWER]: Advancing to term 2
I20260811 11:19:17.760353 14289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:19:17.760490 14289 leader_election.cc:302] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f10b1ad86643dc8e65abe9a6f9e7dd; no voters:
I20260811 11:19:17.760715 14289 leader_election.cc:288] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 11:19:17.760761 14293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 2 FOLLOWER]: Leader election won for term 2
I20260811 11:19:17.760941 14293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 2 LEADER]: Becoming Leader. State: Replica: 74f10b1ad86643dc8e65abe9a6f9e7dd, State: Running, Role: LEADER
I20260811 11:19:17.761096 14293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:19:17.761188 14289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:19:17.761545 14294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } } }
I20260811 11:19:17.761621 14294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: This master's current role is: LEADER
I20260811 11:19:17.761857 14295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f10b1ad86643dc8e65abe9a6f9e7dd. Latest consensus state: current_term: 2 leader_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } } }
I20260811 11:19:17.761969 14295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: This master's current role is: LEADER
I20260811 11:19:17.762130 14298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:19:17.763074 14298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:19:17.763244 14298 catalog_manager.cc:1259] Loaded cluster ID: 489d82000381492fae9cb430f574f011
I20260811 11:19:17.763468 14298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:19:17.764361 14298 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:19:17.764499 14298 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Loaded TSK: 0
I20260811 11:19:17.771823 14298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Generated new TSK 1
I20260811 11:19:17.772012 14298 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 11:19:17.860299 14291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:17.860589 14291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:17.860637 14291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:17.862715 14291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:17.862818 14291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:17.862875 14291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:19:17.864744 14291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:44075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=43283
--tserver_master_addrs=127.28.25.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:17.865072 14291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:17.865361 14291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:17.871595 14317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:17.871708 14320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:17.871628 14318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:17.871923 14291 server_base.cc:1034] running on GCE node
I20260811 11:19:17.872275 14291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:17.872567 14291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:17.873809 14291 hybrid_clock.cc:648] HybridClock initialized: now 1786447157873792 us; error 31 us; skew 500 ppm
Aug 11 11:19:17 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:17.876883 14291 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:19:17.877512 14291 webserver.cc:460] Webserver started at http://127.28.25.1:43283/ using document root <none> and password file <none>
I20260811 11:19:17.877784 14291 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:17.877892 14291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:17.879727 14291 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 11:19:17.880599 14328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:17.880874 14291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:19:17.880995 14291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "efbc2cdc45804ab3b5ba931af9f808f7"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "9b32f3c9a6ae4cc017b85eac7bdbc65c"
server_key_iv: "bc128e0eb26d5cff8f3c61d062664d86"
server_key_version: "encryptionkey@0"
I20260811 11:19:17.881464 14291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:17.901727 14291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:17.904595 14291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:17.904752 14291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:17.905019 14291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:17.905522 14335 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 11:19:17.907450 14291 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 11:19:17.907523 14291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260811 11:19:17.907559 14291 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 11:19:17.908442 14291 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 11:19:17.908488 14291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260811 11:19:17.908591 14335 tablet_bootstrap.cc:492] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Bootstrap starting.
I20260811 11:19:17.916458 14291 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:44075
I20260811 11:19:17.917060 14291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:19:17.919183 14453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:44075 every 8 connection(s)
I20260811 11:19:17.922811 14335 log.cc:824] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:17.926994 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 14291
I20260811 11:19:17.927126 14335 tablet_bootstrap.cc:492] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:19:17.927374 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:44051
--local_ip_for_outbound_sockets=127.28.25.2
--tserver_master_addrs=127.28.25.62:33593
--webserver_port=38377
--webserver_interface=127.28.25.2
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:19:17.927520 14335 tablet_bootstrap.cc:492] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Bootstrap complete.
Aug 11 11:19:17 dist-test-slave-2w3w krb5kdc[13621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:19:17.928996 14335 ts_tablet_manager.cc:1397] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20260811 11:19:17.930167 14335 raft_consensus.cc:348] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:17.930490 14335 raft_consensus.cc:729] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc2cdc45804ab3b5ba931af9f808f7, State: Initialized, Role: FOLLOWER
I20260811 11:19:17.930680 14335 consensus_queue.cc:260] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:17.931066 14335 ts_tablet_manager.cc:1428] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260811 11:19:17.932530 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:17.917994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:54077 (local address 127.28.25.62:33593)
0811 11:19:17.919148 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:17.919162 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:17.919773 (+ 611us) server_negotiation.cc:407] Connection header received
0811 11:19:17.920581 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:17.920592 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:17.920675 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:17.920896 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:17.921829 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:17.923381 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:17.924631 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:17.925298 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:17.925438 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:17.928204 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:17.928208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:17.928208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:17.928227 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:17.930320 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:17.930897 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:17.930899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:17.930900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:17.931106 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:17.931462 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:17.931463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:17.931463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:17.931698 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:17.931767 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:17.932232 (+ 465us) server_negotiation.cc:299] Negotiation successful
0811 11:19:17.932377 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":104,"threads_started":1}
I20260811 11:19:17.932617 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:17.917884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:54077)
0811 11:19:17.919550 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:17.919597 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:17.920290 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:17.920936 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:17.920941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:17.921338 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:17.921641 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:17.921649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:17.923543 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:17.923544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:17.924518 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:17.924520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:17.925457 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:17.925458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:17.925604 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:17.926289 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:19:17.926300 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:17.928075 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:17.930438 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:17.930445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:17.930448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:17.930801 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:17.931218 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:17.931219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:17.931220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:17.931383 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:17.931780 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:17.931797 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:17.931987 (+ 190us) client_negotiation.cc:769] Sending connection context
0811 11:19:17.932124 (+ 137us) client_negotiation.cc:240] Negotiation successful
0811 11:19:17.932339 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1621,"thread_start_us":135,"threads_started":1}
I20260811 11:19:17.933566 14454 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33593
I20260811 11:19:17.933687 14454 heartbeater.cc:464] Registering TS with master...
I20260811 11:19:17.933920 14454 heartbeater.cc:511] Master 127.28.25.62:33593 requested a full tablet report, sending...
I20260811 11:19:17.934918 14251 ts_manager.cc:194] Registered new tserver with Master: efbc2cdc45804ab3b5ba931af9f808f7 (127.28.25.1:44075)
I20260811 11:19:17.936282 14251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:54077
I20260811 11:19:17.937587 14454 heartbeater.cc:503] Master 127.28.25.62:33593 was elected leader, sending a full tablet report...
I20260811 11:19:17.943490 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:17.935666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:44075 (local address 127.0.0.1:44312)
0811 11:19:17.935946 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:17.935979 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:17.936048 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:17.939364 (+ 3316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:17.939368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:17.939371 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:17.939453 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:17.939456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:17.941084 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:17.941085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:17.942034 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:17.942038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:17.943142 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:17.943143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:17.943229 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:17.943243 (+ 14us) client_negotiation.cc:769] Sending connection context
0811 11:19:17.943352 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 11:19:17.943413 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":132,"threads_started":1}
I20260811 11:19:17.943583 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:17.935788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.28.25.1:44075)
0811 11:19:17.939121 (+ 3333us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:17.939127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:17.939143 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:19:17.939190 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:17.939193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:17.939213 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:17.939325 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:17.939547 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:17.940949 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:17.942146 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:17.943047 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:17.943133 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:17.943201 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:17.943446 (+ 245us) server_negotiation.cc:299] Negotiation successful
0811 11:19:17.943496 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3276,"thread_start_us":113,"threads_started":1}
I20260811 11:19:17.944042 14382 tablet_service.cc:1514] Processing DeleteTablet for tablet adcac0b1611a4461b24504383808cd5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:18:57 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:44312
I20260811 11:19:17.944634 14463 tablet_replica.cc:330] stopping tablet replica
I20260811 11:19:17.944769 14463 raft_consensus.cc:2229] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:19:17.944868 14463 raft_consensus.cc:2258] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:19:17.945199 14463 ts_tablet_manager.cc:1905] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:19:17.947726 14463 ts_tablet_manager.cc:1918] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:19:17.947898 14463 log.cc:1196] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal/wals/adcac0b1611a4461b24504383808cd5b
I20260811 11:19:17.948159 14463 ts_tablet_manager.cc:1939] T adcac0b1611a4461b24504383808cd5b P efbc2cdc45804ab3b5ba931af9f808f7: Deleting consensus metadata
I20260811 11:19:17.948841 14233 catalog_manager.cc:4841] TS efbc2cdc45804ab3b5ba931af9f808f7 (127.28.25.1:44075): tablet adcac0b1611a4461b24504383808cd5b (table test-table [id=24b8ae657793441aa7639252bae3b69d]) successfully deleted
W20260811 11:19:18.026600 14459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:18.026914 14459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:18.026994 14459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:18.028554 14459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:18.028659 14459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:18.028713 14459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:19:18.030565 14459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:44051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=38377
--tserver_master_addrs=127.28.25.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:18.030894 14459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:18.031224 14459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:18.037321 14470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:18.037351 14459 server_base.cc:1034] running on GCE node
W20260811 11:19:18.037266 14472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:18.037266 14469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:18.037731 14459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:18.037983 14459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:18.039167 14459 hybrid_clock.cc:648] HybridClock initialized: now 1786447158039150 us; error 49 us; skew 500 ppm
Aug 11 11:19:18 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:18.042028 14459 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:19:18.042622 14459 webserver.cc:460] Webserver started at http://127.28.25.2:38377/ using document root <none> and password file <none>
I20260811 11:19:18.042925 14459 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:18.043056 14459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:18.044756 14459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:19:18.045578 14479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:18.045825 14459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:19:18.045903 14459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "deefe38bc52947db99da0880d75201f7"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "11128e479836bdcffde7ea233a49c7f9"
server_key_iv: "cd4815d63c63941536fcff6d0bfe24b5"
server_key_version: "encryptionkey@0"
I20260811 11:19:18.046284 14459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:18.054929 14459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:18.057216 14459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:18.057360 14459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:18.057565 14459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:18.057904 14459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:18.057937 14459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:18.057986 14459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:18.058012 14459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:18.065268 14459 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:44051
I20260811 11:19:18.065310 14606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:44051 every 8 connection(s)
I20260811 11:19:18.065718 14459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
Aug 11 11:19:18 dist-test-slave-2w3w krb5kdc[13621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:19:18.075786 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 14459
I20260811 11:19:18.076133 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:46095
--local_ip_for_outbound_sockets=127.28.25.3
--tserver_master_addrs=127.28.25.62:33593
--webserver_port=36649
--webserver_interface=127.28.25.3
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:19:18.076248 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.066550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:51939 (local address 127.28.25.62:33593)
0811 11:19:18.066597 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:18.066599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:18.066876 (+ 277us) server_negotiation.cc:407] Connection header received
0811 11:19:18.067543 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:18.067545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:18.067563 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:18.067618 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:18.068700 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.069223 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.070300 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.070529 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.070623 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.073157 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:18.073158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:18.073159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:18.073167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:18.074540 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:18.075070 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:18.075071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:18.075071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:18.075129 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:18.075368 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:18.075369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:18.075369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:18.075503 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:18.075554 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:18.076049 (+ 495us) server_negotiation.cc:299] Negotiation successful
0811 11:19:18.076139 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 11:19:18.076471 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.066513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:51939)
0811 11:19:18.066715 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:18.066773 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:18.067419 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:18.067698 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:18.067703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:18.068349 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:18.068579 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.068585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.069339 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.069340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.070197 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.070199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.070641 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.070643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.070782 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.071423 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:19:18.071432 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:18.073043 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:18.074632 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:18.074641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:18.074643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:18.074949 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:18.075193 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:18.075194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:18.075194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:18.075305 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:18.075600 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:18.075608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:18.075853 (+ 245us) client_negotiation.cc:769] Sending connection context
0811 11:19:18.075999 (+ 146us) client_negotiation.cc:240] Negotiation successful
0811 11:19:18.076200 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260811 11:19:18.076988 14607 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33593
I20260811 11:19:18.077114 14607 heartbeater.cc:464] Registering TS with master...
I20260811 11:19:18.077318 14607 heartbeater.cc:511] Master 127.28.25.62:33593 requested a full tablet report, sending...
I20260811 11:19:18.077764 14251 ts_manager.cc:194] Registered new tserver with Master: deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051)
I20260811 11:19:18.078286 14251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:51939
W20260811 11:19:18.164230 14610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:18.164472 14610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:18.164501 14610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:18.165697 14610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:18.165740 14610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:18.165767 14610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:19:18.167598 14610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:46095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=36649
--tserver_master_addrs=127.28.25.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:18.167841 14610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:18.168056 14610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:18.173775 14616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:18.173851 14618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:18.173887 14610 server_base.cc:1034] running on GCE node
W20260811 11:19:18.174086 14615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:18.174335 14610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:18.174640 14610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:18.175851 14610 hybrid_clock.cc:648] HybridClock initialized: now 1786447158175812 us; error 62 us; skew 500 ppm
Aug 11 11:19:18 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:18.178747 14610 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:19:18.179332 14610 webserver.cc:460] Webserver started at http://127.28.25.3:36649/ using document root <none> and password file <none>
I20260811 11:19:18.179594 14610 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:18.179694 14610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:18.181267 14610 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 11:19:18.181949 14626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:18.182144 14610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:19:18.182257 14610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "b1a9983e15044ca098da92be7c1e0592"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "6b0aa9f6b16cc638113bc9ea4e6a59d4"
server_key_iv: "7a335d0638ff06ab876c393447d22a61"
server_key_version: "encryptionkey@0"
I20260811 11:19:18.182674 14610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:18.193744 14610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:18.196177 14610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:18.196389 14610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:18.196661 14610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:18.197002 14610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:18.197058 14610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:18.197095 14610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:18.197142 14610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:18.203949 14610 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:46095
I20260811 11:19:18.204028 14749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:46095 every 8 connection(s)
I20260811 11:19:18.204394 14610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
Aug 11 11:19:18 dist-test-slave-2w3w krb5kdc[13621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:19:18.213383 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 14610
I20260811 11:19:18.216285 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.205187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:45529 (local address 127.28.25.62:33593)
0811 11:19:18.205263 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:18.205265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:18.205577 (+ 312us) server_negotiation.cc:407] Connection header received
0811 11:19:18.206600 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:18.206602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:18.206614 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:18.206686 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:18.207558 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.208137 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.209217 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.209423 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.209504 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.212265 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:18.212267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:18.212268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:18.212276 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:18.214551 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:18.215135 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:18.215137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:18.215138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:18.215221 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:18.215517 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:18.215518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:18.215518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:18.215642 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:18.215725 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:18.216081 (+ 356us) server_negotiation.cc:299] Negotiation successful
0811 11:19:18.216185 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:19:18.216536 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.205196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:45529)
0811 11:19:18.205421 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:18.205459 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:18.206475 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:18.206771 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:18.206775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:18.207180 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:18.207396 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.207401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.208253 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.208254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.209116 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.209119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.209516 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.209517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.209640 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.210230 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:19:18.210242 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:18.212132 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:18.214641 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:18.214653 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:18.214655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:18.215036 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:18.215280 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:18.215281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:18.215281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:18.215435 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:18.215715 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:18.215724 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:18.215933 (+ 209us) client_negotiation.cc:769] Sending connection context
0811 11:19:18.216043 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 11:19:18.216262 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260811 11:19:18.217119 14750 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33593
I20260811 11:19:18.217257 14750 heartbeater.cc:464] Registering TS with master...
I20260811 11:19:18.217473 14750 heartbeater.cc:511] Master 127.28.25.62:33593 requested a full tablet report, sending...
I20260811 11:19:18.217926 14251 ts_manager.cc:194] Registered new tserver with Master: b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095)
I20260811 11:19:18.218305 14251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:45529
I20260811 11:19:18.221531 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.213968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.28.25.62:33593)
0811 11:19:18.214142 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:18.214144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:18.214241 (+ 97us) server_negotiation.cc:407] Connection header received
0811 11:19:18.214316 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:18.214317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:18.214327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:18.214420 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:18.215256 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.215788 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.216839 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.217175 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.217268 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.218731 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:18.218733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:18.218734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:18.218742 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:18.220273 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:18.220663 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:18.220664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:18.220665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:18.220738 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:18.220957 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:18.220958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:18.220958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:18.221081 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:18.221287 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:18.221325 (+ 38us) server_negotiation.cc:299] Negotiation successful
0811 11:19:18.221415 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
I20260811 11:19:18.222407 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:19:18.225349 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.222865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.28.25.62:33593)
0811 11:19:18.222922 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:18.222923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:18.223104 (+ 181us) server_negotiation.cc:407] Connection header received
0811 11:19:18.223156 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:18.223157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:18.223188 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:18.223251 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:19:18.223453 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.223927 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.224716 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.224917 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.224973 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.225114 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:19:18.225166 (+ 52us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0811 11:19:18.225297 (+ 131us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39690: token expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260811 11:19:18.225414 14753 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39690: token expired
W20260811 11:19:18.225692 14174 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.25.62:33593)
I20260811 11:19:18.232607 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.226092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.28.25.62:33593)
0811 11:19:18.226150 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:18.226152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:18.226159 (+ 7us) server_negotiation.cc:407] Connection header received
0811 11:19:18.226177 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:18.226177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:18.226185 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:18.226286 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:18.226885 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.227446 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.228193 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.228395 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.228470 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.229898 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:18.229900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:18.229900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:18.229908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:18.231318 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:18.231727 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:18.231729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:18.231729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:18.231793 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:18.232009 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:18.232010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:18.232010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:18.232176 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:18.232288 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:18.232404 (+ 116us) server_negotiation.cc:299] Negotiation successful
0811 11:19:18.232497 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 11:19:18.233974 14251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:19:18.235775 14251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:19:18.238372 14382 tablet_service.cc:1467] Processing CreateTablet for tablet 39f5dc8ca16f48889b3d792bc65cd23b (DEFAULT_TABLE table=test-table [id=119936ea650f47db9aaf06a88bff2582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:19:18.238620 14382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f5dc8ca16f48889b3d792bc65cd23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:18.240024 14335 tablet_bootstrap.cc:492] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: Bootstrap starting.
I20260811 11:19:18.240525 14335 tablet_bootstrap.cc:654] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:18.241117 14335 tablet_bootstrap.cc:492] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: No bootstrap required, opened a new log
I20260811 11:19:18.241173 14335 ts_tablet_manager.cc:1397] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:18.241288 14335 raft_consensus.cc:348] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:18.241384 14335 raft_consensus.cc:374] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:18.241451 14335 raft_consensus.cc:729] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc2cdc45804ab3b5ba931af9f808f7, State: Initialized, Role: FOLLOWER
I20260811 11:19:18.241501 14335 consensus_queue.cc:260] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:18.241602 14335 ts_tablet_manager.cc:1428] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 11:19:18.244107 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.240017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.28.25.3:46095)
0811 11:19:18.240303 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:18.240308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:18.240321 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:19:18.240402 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:18.240404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:18.240423 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:18.240558 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:18.240666 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.241862 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.242949 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.243715 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.243802 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.243869 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:18.243951 (+ 82us) server_negotiation.cc:299] Negotiation successful
0811 11:19:18.243992 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":147,"threads_started":1}
I20260811 11:19:18.244112 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.238109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.28.25.2:44051)
0811 11:19:18.238312 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:18.238318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:18.240439 (+ 2121us) server_negotiation.cc:407] Connection header received
0811 11:19:18.240531 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:18.240534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:18.240550 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:18.240624 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:18.240890 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.241996 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.242985 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.243731 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.243817 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.243877 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:18.243983 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 11:19:18.244022 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260811 11:19:18.244103 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.237917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:46095 (local address 127.0.0.1:58158)
0811 11:19:18.238067 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:18.238078 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:18.238140 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:18.240553 (+ 2413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:18.240554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:18.240555 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:18.240611 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.240614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.241993 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.241994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.242822 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.242825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.243800 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.243801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.243868 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.243872 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:19:18.243933 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 11:19:18.243991 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":6}
I20260811 11:19:18.244113 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.237917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:44051 (local address 127.0.0.1:60226)
0811 11:19:18.240337 (+ 2420us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:18.240349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:18.240449 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:18.240728 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:18.240729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:18.240730 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:18.240766 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.240767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.242079 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.242081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.242825 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.242827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.243833 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.243834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.243908 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.243912 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:19:18.243969 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 11:19:18.244019 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2406,"mutex_wait_us":2316,"thread_start_us":64,"threads_started":1}
I20260811 11:19:18.245240 14532 tablet_service.cc:1467] Processing CreateTablet for tablet 39f5dc8ca16f48889b3d792bc65cd23b (DEFAULT_TABLE table=test-table [id=119936ea650f47db9aaf06a88bff2582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:19:18.245277 14680 tablet_service.cc:1467] Processing CreateTablet for tablet 39f5dc8ca16f48889b3d792bc65cd23b (DEFAULT_TABLE table=test-table [id=119936ea650f47db9aaf06a88bff2582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:19:18.245604 14680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f5dc8ca16f48889b3d792bc65cd23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:18.245604 14532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f5dc8ca16f48889b3d792bc65cd23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:18.246438 14460 raft_consensus.cc:482] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:19:18.246564 14460 raft_consensus.cc:504] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:18.246932 14460 leader_election.cc:288] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051), b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095)
I20260811 11:19:18.248934 14761 tablet_bootstrap.cc:492] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: Bootstrap starting.
I20260811 11:19:18.249812 14761 tablet_bootstrap.cc:654] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:18.250558 14761 log.cc:824] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:18.250979 14760 tablet_bootstrap.cc:492] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: Bootstrap starting.
I20260811 11:19:18.251523 14761 tablet_bootstrap.cc:492] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: No bootstrap required, opened a new log
I20260811 11:19:18.251585 14761 ts_tablet_manager.cc:1397] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:19:18.252602 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.249214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:52327 (local address 127.28.25.3:46095)
0811 11:19:18.249249 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:18.249251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:18.249430 (+ 179us) server_negotiation.cc:407] Connection header received
0811 11:19:18.249485 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:18.249486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:18.249496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:18.249548 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:18.250038 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.250651 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.251846 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.252240 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.252323 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.252333 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:18.252493 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 11:19:18.252530 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260811 11:19:18.252635 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.249166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:46095 (local address 127.28.25.1:52327)
0811 11:19:18.249350 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:18.249361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:18.249421 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:18.249550 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:18.249551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:18.249553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:18.249972 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.249976 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.250725 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.250726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.251756 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.251759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.252334 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.252335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.252419 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.252423 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:19:18.252487 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 11:19:18.252529 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":126,"threads_started":1}
I20260811 11:19:18.253317 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.249541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:57879 (local address 127.28.25.2:44051)
0811 11:19:18.249586 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:18.249588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:18.249596 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:19:18.249618 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:18.249619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:18.249628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:18.249680 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:18.250247 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.250930 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.252332 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.252889 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.252939 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.252949 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:18.253199 (+ 250us) server_negotiation.cc:299] Negotiation successful
0811 11:19:18.253255 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 11:19:18.253571 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.247119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:44051 (local address 127.28.25.1:57879)
0811 11:19:18.249348 (+ 2229us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:18.249358 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:18.249423 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:18.250061 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:18.250062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:18.250064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:18.250155 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.250175 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.251063 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.251064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.252210 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:18.252212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.253037 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.253038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:18.253113 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.253116 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:19:18.253442 (+ 326us) client_negotiation.cc:240] Negotiation successful
0811 11:19:18.253489 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2210}
I20260811 11:19:18.252949 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f5dc8ca16f48889b3d792bc65cd23b" candidate_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a9983e15044ca098da92be7c1e0592" is_pre_election: true
I20260811 11:19:18.253877 14761 raft_consensus.cc:348] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:18.254074 14761 raft_consensus.cc:374] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:18.254118 14761 raft_consensus.cc:729] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deefe38bc52947db99da0880d75201f7, State: Initialized, Role: FOLLOWER
I20260811 11:19:18.254249 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f5dc8ca16f48889b3d792bc65cd23b" candidate_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deefe38bc52947db99da0880d75201f7" is_pre_election: true
I20260811 11:19:18.254295 14761 consensus_queue.cc:260] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:18.254420 14553 raft_consensus.cc:2454] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbc2cdc45804ab3b5ba931af9f808f7 in term 0.
I20260811 11:19:18.254787 14761 ts_tablet_manager.cc:1428] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:19:18.254815 14330 leader_election.cc:302] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: deefe38bc52947db99da0880d75201f7, efbc2cdc45804ab3b5ba931af9f808f7; no voters:
I20260811 11:19:18.254940 14460 raft_consensus.cc:2781] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:19:18.254993 14607 heartbeater.cc:503] Master 127.28.25.62:33593 was elected leader, sending a full tablet report...
I20260811 11:19:18.255025 14460 raft_consensus.cc:482] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:19:18.255050 14460 raft_consensus.cc:3037] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 0 FOLLOWER]: Advancing to term 1
W20260811 11:19:18.255285 14332 leader_election.cc:341] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095): Illegal state: must be running to vote when last-logged opid is not known
I20260811 11:19:18.255511 14760 tablet_bootstrap.cc:654] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:18.255848 14760 log.cc:824] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:18.256012 14460 raft_consensus.cc:504] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:18.256182 14460 leader_election.cc:288] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 1 election: Requested vote from peers deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051), b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095)
I20260811 11:19:18.256366 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f5dc8ca16f48889b3d792bc65cd23b" candidate_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a9983e15044ca098da92be7c1e0592"
I20260811 11:19:18.256373 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f5dc8ca16f48889b3d792bc65cd23b" candidate_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deefe38bc52947db99da0880d75201f7"
I20260811 11:19:18.256459 14553 raft_consensus.cc:3037] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7 [term 0 FOLLOWER]: Advancing to term 1
W20260811 11:19:18.256587 14332 leader_election.cc:341] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095): Illegal state: must be running to vote when last-logged opid is not known
I20260811 11:19:18.256770 14760 tablet_bootstrap.cc:492] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: No bootstrap required, opened a new log
I20260811 11:19:18.256845 14760 ts_tablet_manager.cc:1397] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260811 11:19:18.257386 14553 raft_consensus.cc:2454] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbc2cdc45804ab3b5ba931af9f808f7 in term 1.
I20260811 11:19:18.257601 14330 leader_election.cc:302] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: deefe38bc52947db99da0880d75201f7, efbc2cdc45804ab3b5ba931af9f808f7; no voters: b1a9983e15044ca098da92be7c1e0592
I20260811 11:19:18.257696 14460 raft_consensus.cc:2781] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:19:18.257893 14460 raft_consensus.cc:686] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 LEADER]: Becoming Leader. State: Replica: efbc2cdc45804ab3b5ba931af9f808f7, State: Running, Role: LEADER
I20260811 11:19:18.257974 14460 consensus_queue.cc:237] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:18.258328 14760 raft_consensus.cc:348] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:18.258476 14760 raft_consensus.cc:374] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:18.258524 14760 raft_consensus.cc:729] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a9983e15044ca098da92be7c1e0592, State: Initialized, Role: FOLLOWER
I20260811 11:19:18.258675 14760 consensus_queue.cc:260] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:18.258744 14251 catalog_manager.cc:5495] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to efbc2cdc45804ab3b5ba931af9f808f7 (127.28.25.1). New cstate: current_term: 1 leader_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } health_report { overall_health: HEALTHY } } }
I20260811 11:19:18.258996 14760 ts_tablet_manager.cc:1428] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 11:19:18.259017 14750 heartbeater.cc:503] Master 127.28.25.62:33593 was elected leader, sending a full tablet report...
I20260811 11:19:18.271451 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.268785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.28.25.1:44075)
0811 11:19:18.268909 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:18.268912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:18.268919 (+ 7us) server_negotiation.cc:407] Connection header received
0811 11:19:18.268963 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:18.268964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:18.268987 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:18.269068 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:19:18.269217 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.269717 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.270539 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:18.270746 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:18.270823 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:18.271063 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:19:18.271176 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:19:18.271260 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:18.271305 (+ 45us) server_negotiation.cc:299] Negotiation successful
0811 11:19:18.271332 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 11:19:18.273527 14701 raft_consensus.cc:3037] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:19:18.273588 14553 raft_consensus.cc:1261] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7 [term 1 FOLLOWER]: Refusing update from remote peer efbc2cdc45804ab3b5ba931af9f808f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:19:18.274013 14460 consensus_queue.cc:1035] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:19:18.274804 14701 raft_consensus.cc:1261] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592 [term 1 FOLLOWER]: Refusing update from remote peer efbc2cdc45804ab3b5ba931af9f808f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:19:18.275262 14460 consensus_queue.cc:1035] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:19:18.281329 14250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:19:18.281517 14250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:19:18.282382 14250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Sending DeleteTablet for 3 replicas of tablet 39f5dc8ca16f48889b3d792bc65cd23b
I20260811 11:19:18.282817 14680 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f5dc8ca16f48889b3d792bc65cd23b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:19:18 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:58158
I20260811 11:19:18.282830 14532 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f5dc8ca16f48889b3d792bc65cd23b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:19:18 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:60226
I20260811 11:19:18.283154 14382 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f5dc8ca16f48889b3d792bc65cd23b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:19:18 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:44312
I20260811 11:19:18.283299 14463 tablet_replica.cc:330] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: stopping tablet replica
I20260811 11:19:18.283346 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 14291
I20260811 11:19:18.283398 14463 raft_consensus.cc:2229] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:19:18.283521 14463 raft_consensus.cc:2258] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:19:18.288393 14781 tablet_replica.cc:330] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: stopping tablet replica
I20260811 11:19:18.288607 14781 raft_consensus.cc:2229] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:19:18.288798 14781 raft_consensus.cc:2258] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:19:18.289072 14780 tablet_replica.cc:330] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: stopping tablet replica
I20260811 11:19:18.289191 14780 raft_consensus.cc:2229] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:19:18.289292 14780 raft_consensus.cc:2258] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:19:18.289569 14781 ts_tablet_manager.cc:1905] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:19:18.289799 14780 ts_tablet_manager.cc:1905] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:19:18.291935 14781 ts_tablet_manager.cc:1918] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:19:18.292052 14781 log.cc:1196] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/39f5dc8ca16f48889b3d792bc65cd23b
I20260811 11:19:18.292009 14780 ts_tablet_manager.cc:1918] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:19:18.292155 14780 log.cc:1196] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/39f5dc8ca16f48889b3d792bc65cd23b
I20260811 11:19:18.292395 14781 ts_tablet_manager.cc:1939] T 39f5dc8ca16f48889b3d792bc65cd23b P b1a9983e15044ca098da92be7c1e0592: Deleting consensus metadata
I20260811 11:19:18.292546 14780 ts_tablet_manager.cc:1939] T 39f5dc8ca16f48889b3d792bc65cd23b P deefe38bc52947db99da0880d75201f7: Deleting consensus metadata
I20260811 11:19:18.292970 14236 catalog_manager.cc:4841] TS b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095): tablet 39f5dc8ca16f48889b3d792bc65cd23b (table test-table [id=119936ea650f47db9aaf06a88bff2582]) successfully deleted
I20260811 11:19:18.293032 14234 catalog_manager.cc:4841] TS deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051): tablet 39f5dc8ca16f48889b3d792bc65cd23b (table test-table [id=119936ea650f47db9aaf06a88bff2582]) successfully deleted
W20260811 11:19:18.295001 14233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:44075 (error 108)
I20260811 11:19:18.295645 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:18.295475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:44075 (local address 127.0.0.1:44322)
0811 11:19:18.295554 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:18.295600 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:44075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 11:19:18.295804 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 14459
W20260811 11:19:18.295811 14233 catalog_manager.cc:4584] TS efbc2cdc45804ab3b5ba931af9f808f7 (127.28.25.1:44075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39f5dc8ca16f48889b3d792bc65cd23b: Network error: Client connection negotiation failed: client connection to 127.28.25.1:44075: connect: Connection refused (error 111)
I20260811 11:19:18.304586 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 14610
I20260811 11:19:18.313526 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 14215
I20260811 11:19:38.321871 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33593
--webserver_interface=127.28.25.62
--webserver_port=46553
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:33593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:19:38.406762 14784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:38.407065 14784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:38.407097 14784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:38.408362 14784 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 11:19:38.408409 14784 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 11:19:38.408429 14784 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 11:19:38.408442 14784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:19:38.408455 14784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:38.408505 14784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:38.408524 14784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:19:38.408546 14784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:19:38.410172 14784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:33593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=46553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:38.410458 14784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:38.410725 14784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:38.416071 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:38.416066 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:38.416059 14793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:38.416087 14784 server_base.cc:1034] running on GCE node
I20260811 11:19:38.416682 14784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:38.417022 14784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:38.418236 14784 hybrid_clock.cc:648] HybridClock initialized: now 1786447178418220 us; error 43 us; skew 500 ppm
Aug 11 11:19:38 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:38.427259 14784 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:19:38.427872 14784 webserver.cc:460] Webserver started at http://127.28.25.62:46553/ using document root <none> and password file <none>
I20260811 11:19:38.428164 14784 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:38.428241 14784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:38.429734 14784 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:19:38.430526 14801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:38.430747 14784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:38.430856 14784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd"
format_stamp: "Formatted at 2026-08-11 11:18:56 on dist-test-slave-2w3w"
server_key: "bc8ea00b4f5d1594f9bec30b61a8c9e5"
server_key_iv: "6b7652cde7038f004bc2e10753fe4843"
server_key_version: "encryptionkey@0"
I20260811 11:19:38.431325 14784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:38.453872 14784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:38.456192 14784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:38.456372 14784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:38.460913 14784 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:33593
I20260811 11:19:38.460956 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:33593 every 8 connection(s)
I20260811 11:19:38.461374 14784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:19:38.463117 14858 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:19:38.463888 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Bootstrap starting.
I20260811 11:19:38.468672 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 14784
I20260811 11:19:38.469058 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:44075
--local_ip_for_outbound_sockets=127.28.25.1
--tserver_master_addrs=127.28.25.62:33593
--webserver_port=43283
--webserver_interface=127.28.25.1
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:19:38.474419 14858 log.cc:824] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:38.477183 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:19:38.477567 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Bootstrap complete.
I20260811 11:19:38.480189 14858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:19:38.480492 14858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f10b1ad86643dc8e65abe9a6f9e7dd, State: Initialized, Role: FOLLOWER
I20260811 11:19:38.480688 14858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:19:38.480770 14858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:19:38.480826 14858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:19:38.480882 14858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 2 FOLLOWER]: Advancing to term 3
I20260811 11:19:38.482005 14858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:19:38.482141 14858 leader_election.cc:302] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f10b1ad86643dc8e65abe9a6f9e7dd; no voters:
I20260811 11:19:38.482373 14858 leader_election.cc:288] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 11:19:38.482419 14862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 3 FOLLOWER]: Leader election won for term 3
I20260811 11:19:38.482544 14862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [term 3 LEADER]: Becoming Leader. State: Replica: 74f10b1ad86643dc8e65abe9a6f9e7dd, State: Running, Role: LEADER
I20260811 11:19:38.482676 14858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:19:38.482709 14862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } }
I20260811 11:19:38.483048 14864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } } }
I20260811 11:19:38.483247 14864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: This master's current role is: LEADER
I20260811 11:19:38.483042 14862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f10b1ad86643dc8e65abe9a6f9e7dd. Latest consensus state: current_term: 3 leader_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f10b1ad86643dc8e65abe9a6f9e7dd" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33593 } } }
I20260811 11:19:38.483650 14873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:19:38.483749 14862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd [sys.catalog]: This master's current role is: LEADER
I20260811 11:19:38.484705 14873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:19:38.484920 14873 catalog_manager.cc:1259] Loaded cluster ID: 489d82000381492fae9cb430f574f011
I20260811 11:19:38.485132 14873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:19:38.486395 14873 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:19:38.486645 14873 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Loaded TSK: 0
I20260811 11:19:38.486727 14873 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74f10b1ad86643dc8e65abe9a6f9e7dd: Loaded TSK: 1
F20260811 11:19:38.487040 14873 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f74736139ad google::LogMessage::Fail() at ??:0
@ 0x7f747361688a google::LogMessage::SendToLog() at ??:0
@ 0x7f74736134de google::LogMessage::Flush() at ??:0
@ 0x7f7473614a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7475d80104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7475d7dd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f747b56d1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f747b56d48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f747b56d594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f747b533c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f747b53dad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f74754e10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f74754d6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f747633a6db start_thread at ??:0
@ 0x7f747211161f clone at ??:0
W20260811 11:19:38.578668 14860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:38.578912 14860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:38.578939 14860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:38.580269 14860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:38.580325 14860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:38.580358 14860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:19:38.581938 14860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:44075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=43283
--tserver_master_addrs=127.28.25.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:38.582207 14860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:38.582472 14860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:38.588162 14902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:38.588259 14905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:38.588177 14903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:38.588455 14860 server_base.cc:1034] running on GCE node
I20260811 11:19:38.588723 14860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:38.589046 14860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:38.590236 14860 hybrid_clock.cc:648] HybridClock initialized: now 1786447178590219 us; error 34 us; skew 500 ppm
Aug 11 11:19:38 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:38.593358 14860 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:19:38.593874 14860 webserver.cc:460] Webserver started at http://127.28.25.1:43283/ using document root <none> and password file <none>
I20260811 11:19:38.594134 14860 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:38.594211 14860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:38.595993 14860 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:19:38.597010 14913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:38.597306 14860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:38.597440 14860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "efbc2cdc45804ab3b5ba931af9f808f7"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "9b32f3c9a6ae4cc017b85eac7bdbc65c"
server_key_iv: "bc128e0eb26d5cff8f3c61d062664d86"
server_key_version: "encryptionkey@0"
I20260811 11:19:38.597892 14860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:38.609522 14860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:38.612049 14860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:38.612236 14860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:38.612504 14860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:38.612990 14920 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 11:19:38.615123 14860 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 11:19:38.615197 14860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260811 11:19:38.615247 14860 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 11:19:38.615993 14860 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 11:19:38.616056 14860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260811 11:19:38.616130 14920 tablet_bootstrap.cc:492] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: Bootstrap starting.
I20260811 11:19:38.624078 14860 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:44075
I20260811 11:19:38.624277 15040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:44075 every 8 connection(s)
I20260811 11:19:38.624574 14860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:19:38.625641 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 14860
I20260811 11:19:38.625972 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:44051
--local_ip_for_outbound_sockets=127.28.25.2
--tserver_master_addrs=127.28.25.62:33593
--webserver_port=38377
--webserver_interface=127.28.25.2
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:19:38.629204 14920 log.cc:824] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:38.632400 14920 tablet_bootstrap.cc:492] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:19:38.632795 14920 tablet_bootstrap.cc:492] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: Bootstrap complete.
I20260811 11:19:38.634299 14920 ts_tablet_manager.cc:1397] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.012s
I20260811 11:19:38.636023 14920 raft_consensus.cc:348] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:38.636442 14920 raft_consensus.cc:729] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc2cdc45804ab3b5ba931af9f808f7, State: Initialized, Role: FOLLOWER
I20260811 11:19:38.636626 14920 consensus_queue.cc:260] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:38.636927 14920 ts_tablet_manager.cc:1428] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.003s
W20260811 11:19:38.707015 15043 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:19:38.625331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:55185)
0811 11:19:38.628876 (+ 3545us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:38.628940 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:38.629830 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:38.706754 (+ 76924us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3484,"thread_start_us":129,"threads_started":1}
W20260811 11:19:38.707306 15041 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:33593 (0 consecutive failures): Network error: Failed to ping master at 127.28.25.62:33593: Client connection negotiation failed: client connection to 127.28.25.62:33593: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 11:19:38.707710 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:38.707561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:52197)
0811 11:19:38.707621 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:38.707675 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 11:19:38.708083 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:38.707974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:33191)
0811 11:19:38.708026 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:38.708058 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260811 11:19:38.708250 15041 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:19:38.717415 15044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:38.717643 15044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:38.717669 15044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:38.718806 15044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:38.718847 15044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:38.718873 15044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:19:38.720902 15044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:44051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=38377
--tserver_master_addrs=127.28.25.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:38.721134 15044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:38.721328 15044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:38.726536 15054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:38.726557 15051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:38.726545 15052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:38.726807 15044 server_base.cc:1034] running on GCE node
I20260811 11:19:38.727115 15044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:38.727383 15044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:38.728554 15044 hybrid_clock.cc:648] HybridClock initialized: now 1786447178728523 us; error 50 us; skew 500 ppm
Aug 11 11:19:38 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:38.731246 15044 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:19:38.731746 15044 webserver.cc:460] Webserver started at http://127.28.25.2:38377/ using document root <none> and password file <none>
I20260811 11:19:38.732056 15044 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:38.732110 15044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:38.733528 15044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:19:38.734304 15062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:38.734520 15044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:38.734625 15044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "deefe38bc52947db99da0880d75201f7"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "11128e479836bdcffde7ea233a49c7f9"
server_key_iv: "cd4815d63c63941536fcff6d0bfe24b5"
server_key_version: "encryptionkey@0"
I20260811 11:19:38.735136 15044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:38.747246 15044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:38.749475 15044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:38.749651 15044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:38.749919 15044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:38.750242 15044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:38.750303 15044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:38.750348 15044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:38.750382 15044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:38.756768 15044 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:44051
I20260811 11:19:38.756811 15190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:44051 every 8 connection(s)
I20260811 11:19:38.757184 15044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:19:38.758358 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:38.757917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:52269)
0811 11:19:38.758107 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:38.758189 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
W20260811 11:19:38.758817 15191 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:33593 (0 consecutive failures): Network error: Failed to ping master at 127.28.25.62:33593: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
I20260811 11:19:38.759164 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:38.759066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:33621)
0811 11:19:38.759113 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:38.759143 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260811 11:19:38.759527 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:38.759444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:35391)
0811 11:19:38.759489 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:38.759512 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260811 11:19:38.759649 15191 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 11:19:38.763545 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 15044
I20260811 11:19:38.763928 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:46095
--local_ip_for_outbound_sockets=127.28.25.3
--tserver_master_addrs=127.28.25.62:33593
--webserver_port=36649
--webserver_interface=127.28.25.3
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:19:38.853062 15194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:38.853323 15194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:38.853353 15194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:38.854655 15194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:38.854727 15194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:38.854758 15194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:19:38.856451 15194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:46095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=36649
--tserver_master_addrs=127.28.25.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:38.856750 15194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:38.857018 15194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:38.862870 15200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:38.862937 15203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:38.862891 15199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:38.863159 15194 server_base.cc:1034] running on GCE node
I20260811 11:19:38.863483 15194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:38.863847 15194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:38.865046 15194 hybrid_clock.cc:648] HybridClock initialized: now 1786447178865020 us; error 39 us; skew 500 ppm
Aug 11 11:19:38 dist-test-slave-2w3w krb5kdc[13621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:38.868287 15194 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:19:38.868872 15194 webserver.cc:460] Webserver started at http://127.28.25.3:36649/ using document root <none> and password file <none>
I20260811 11:19:38.869136 15194 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:38.869217 15194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:38.870819 15194 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:19:38.871635 15210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:38.871868 15194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:19:38.871990 15194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "b1a9983e15044ca098da92be7c1e0592"
format_stamp: "Formatted at 2026-08-11 11:18:57 on dist-test-slave-2w3w"
server_key: "6b0aa9f6b16cc638113bc9ea4e6a59d4"
server_key_iv: "7a335d0638ff06ab876c393447d22a61"
server_key_version: "encryptionkey@0"
I20260811 11:19:38.872418 15194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:38.885314 15194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:38.887779 15194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:38.887959 15194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:38.888219 15194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:38.888579 15194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:38.888636 15194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:38.888684 15194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:38.888715 15194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:38.895179 15194 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:46095
I20260811 11:19:38.895231 15335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:46095 every 8 connection(s)
I20260811 11:19:38.895610 15194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:19:38.896853 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:38.896338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:43039)
0811 11:19:38.896517 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:38.896646 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
W20260811 11:19:38.897303 15337 heartbeater.cc:644] Failed to heartbeat to 127.28.25.62:33593 (0 consecutive failures): Network error: Failed to ping master at 127.28.25.62:33593: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
I20260811 11:19:38.897624 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:38.897529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:57485)
0811 11:19:38.897574 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:38.897603 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260811 11:19:38.898082 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:38.897984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:45567)
0811 11:19:38.898039 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:38.898065 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 11:19:38.898236 15337 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 11:19:38.900476 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 15194
I20260811 11:19:39.708962 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:39.708683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:36183)
0811 11:19:39.708828 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:39.708874 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260811 11:19:39.760304 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:39.760039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:52875)
0811 11:19:39.760200 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:39.760243 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260811 11:19:39.898939 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:39.898668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:50923)
0811 11:19:39.898828 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:39.898873 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260811 11:19:40.097507 15344 raft_consensus.cc:482] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:19:40.097640 15344 raft_consensus.cc:504] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:40.098151 15344 leader_election.cc:288] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051), b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095)
Aug 11 11:19:40 dist-test-slave-2w3w krb5kdc[13621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.3@KRBTEST.COM
Aug 11 11:19:40 dist-test-slave-2w3w krb5kdc[13621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.2@KRBTEST.COM
I20260811 11:19:40.109027 15345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:40.098398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:46095 (local address 127.28.25.1:40961)
0811 11:19:40.098570 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:40.098579 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:40.098648 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:40.099515 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:40.099517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:40.099911 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:40.100102 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:40.100109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:40.101295 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:40.101297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:40.102116 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:40.102118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:40.102770 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:40.102771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:40.102891 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:40.103581 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:19:40.103585 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:40.105096 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:40.107484 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:40.107491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:40.107493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:40.107708 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:40.107984 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:40.107985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:40.107985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:40.108091 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:40.108486 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:40.108493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:40.108617 (+ 124us) client_negotiation.cc:769] Sending connection context
0811 11:19:40.108727 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 11:19:40.108918 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":89,"thread_start_us":59,"threads_started":1}
I20260811 11:19:40.109041 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:40.098546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:40961 (local address 127.28.25.3:46095)
0811 11:19:40.098719 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:40.098732 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:40.098769 (+ 37us) server_negotiation.cc:407] Connection header received
0811 11:19:40.099274 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:40.099277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:40.099321 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:40.099497 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:40.100247 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:40.101193 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:40.102219 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:40.102660 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:40.102784 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:40.105215 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:40.105220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:40.105220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:40.105233 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:40.107402 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:40.107804 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:40.107808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:40.107808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:40.107909 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:40.108167 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:40.108168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:40.108168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:40.108410 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:40.108506 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:40.108782 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 11:19:40.108923 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":60,"threads_started":1}
I20260811 11:19:40.109081 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:40.098546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:34391 (local address 127.28.25.2:44051)
0811 11:19:40.098742 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:40.098750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:40.098779 (+ 29us) server_negotiation.cc:407] Connection header received
0811 11:19:40.099278 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:40.099280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:40.099320 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:40.099466 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:40.100235 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:40.101267 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:40.102226 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:40.102644 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:40.102804 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:40.105715 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:40.105719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:40.105720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:40.105734 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:40.107651 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:40.107998 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:40.108000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:40.108001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:40.108095 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:40.108313 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:40.108314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:40.108314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:40.108550 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:40.108642 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:40.108806 (+ 164us) server_negotiation.cc:299] Negotiation successful
0811 11:19:40.108964 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260811 11:19:40.109139 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:40.098386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:44051 (local address 127.28.25.1:34391)
0811 11:19:40.098440 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:40.098451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:40.098510 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:40.099513 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:40.099516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:40.099910 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:40.100102 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:40.100109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:40.101364 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:40.101365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:40.102122 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:40.102125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:40.102772 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:40.102773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:40.102889 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:40.103553 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:19:40.103561 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:40.105596 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:40.107730 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:40.107731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:40.107731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:40.107901 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:40.108175 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:40.108176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:40.108176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:40.108244 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:40.108624 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:40.108625 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:40.108671 (+ 46us) client_negotiation.cc:769] Sending connection context
0811 11:19:40.108742 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 11:19:40.109006 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":125}
W20260811 11:19:40.110404 14917 leader_election.cc:341] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095): Not found: Tablet not found: 39f5dc8ca16f48889b3d792bc65cd23b
W20260811 11:19:40.110636 14915 leader_election.cc:341] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051): Not found: Tablet not found: 39f5dc8ca16f48889b3d792bc65cd23b
I20260811 11:19:40.110694 14915 leader_election.cc:302] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efbc2cdc45804ab3b5ba931af9f808f7; no voters: b1a9983e15044ca098da92be7c1e0592, deefe38bc52947db99da0880d75201f7
I20260811 11:19:40.110823 15344 raft_consensus.cc:2726] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:19:40.709862 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:40.709567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:46221)
0811 11:19:40.709751 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:40.709805 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260811 11:19:40.761122 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:40.760865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:58519)
0811 11:19:40.761028 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:40.761075 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260811 11:19:40.899798 15350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:40.899560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:35471)
0811 11:19:40.899704 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:40.899750 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260811 11:19:41.710780 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:41.710415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:60215)
0811 11:19:41.710697 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:41.710730 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":94,"threads_started":1}
I20260811 11:19:41.761925 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:41.761666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:34829)
0811 11:19:41.761826 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:41.761873 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260811 11:19:41.765380 15353 raft_consensus.cc:482] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:19:41.765477 15353 raft_consensus.cc:504] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:41.765699 15353 leader_election.cc:288] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051), b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095)
W20260811 11:19:41.766291 14915 leader_election.cc:341] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051): Not found: Tablet not found: 39f5dc8ca16f48889b3d792bc65cd23b
W20260811 11:19:41.766368 14917 leader_election.cc:341] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095): Not found: Tablet not found: 39f5dc8ca16f48889b3d792bc65cd23b
I20260811 11:19:41.766412 14917 leader_election.cc:302] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efbc2cdc45804ab3b5ba931af9f808f7; no voters: b1a9983e15044ca098da92be7c1e0592, deefe38bc52947db99da0880d75201f7
I20260811 11:19:41.766510 15353 raft_consensus.cc:2726] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:19:41.900656 15354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:41.900325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:56043)
0811 11:19:41.900561 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:41.900608 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":164,"threads_started":1}
I20260811 11:19:42.711689 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:42.711383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:36111)
0811 11:19:42.711580 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:42.711635 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260811 11:19:42.762727 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:42.762487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:43075)
0811 11:19:42.762632 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:42.762677 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260811 11:19:42.901672 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:42.901384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:41019)
0811 11:19:42.901571 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:42.901621 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
I20260811 11:19:43.679225 15358 raft_consensus.cc:482] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:19:43.679342 15358 raft_consensus.cc:504] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:43.679615 15358 leader_election.cc:288] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051), b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095)
W20260811 11:19:43.680069 14915 leader_election.cc:341] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051): Not found: Tablet not found: 39f5dc8ca16f48889b3d792bc65cd23b
W20260811 11:19:43.680172 14917 leader_election.cc:341] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095): Not found: Tablet not found: 39f5dc8ca16f48889b3d792bc65cd23b
I20260811 11:19:43.680275 14917 leader_election.cc:302] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efbc2cdc45804ab3b5ba931af9f808f7; no voters: b1a9983e15044ca098da92be7c1e0592, deefe38bc52947db99da0880d75201f7
I20260811 11:19:43.680389 15358 raft_consensus.cc:2726] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:19:43.712666 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:43.712434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:49773)
0811 11:19:43.712577 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:43.712617 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260811 11:19:43.763540 15360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:43.763283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:51119)
0811 11:19:43.763446 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:43.763493 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260811 11:19:43.902705 15361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:43.902424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:38647)
0811 11:19:43.902595 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:43.902653 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260811 11:19:44.713729 15362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:44.713322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:60165)
0811 11:19:44.713584 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:44.713655 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
I20260811 11:19:44.764360 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:44.764110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:58111)
0811 11:19:44.764262 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:44.764311 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260811 11:19:44.903519 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:44.903266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:42313)
0811 11:19:44.903422 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:44.903468 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260811 11:19:45.714648 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:45.714347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:51779)
0811 11:19:45.714530 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:45.714598 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
I20260811 11:19:45.765179 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:45.764901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:52115)
0811 11:19:45.765084 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:45.765129 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":95,"threads_started":1}
I20260811 11:19:45.904584 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:45.904279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:56679)
0811 11:19:45.904486 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:45.904532 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":120,"threads_started":1}
I20260811 11:19:46.715662 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:46.715364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:45451)
0811 11:19:46.715564 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:46.715611 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":107,"threads_started":1}
I20260811 11:19:46.766119 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:46.765851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:39051)
0811 11:19:46.766025 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:46.766071 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260811 11:19:46.905463 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:46.905162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:49235)
0811 11:19:46.905360 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:46.905410 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260811 11:19:47.716548 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:47.716268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:45213)
0811 11:19:47.716442 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:47.716489 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260811 11:19:47.767045 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:47.766620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:40135)
0811 11:19:47.766860 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:47.766905 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":109,"threads_started":1}
I20260811 11:19:47.906270 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:47.906014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:60811)
0811 11:19:47.906176 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:47.906220 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260811 11:19:48.009655 15376 raft_consensus.cc:482] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:19:48.009757 15376 raft_consensus.cc:504] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deefe38bc52947db99da0880d75201f7" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 44051 } } peers { permanent_uuid: "b1a9983e15044ca098da92be7c1e0592" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 46095 } } peers { permanent_uuid: "efbc2cdc45804ab3b5ba931af9f808f7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44075 } }
I20260811 11:19:48.009946 15376 leader_election.cc:288] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051), b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095)
W20260811 11:19:48.010399 14915 leader_election.cc:341] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer deefe38bc52947db99da0880d75201f7 (127.28.25.2:44051): Not found: Tablet not found: 39f5dc8ca16f48889b3d792bc65cd23b
W20260811 11:19:48.010514 14917 leader_election.cc:341] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1a9983e15044ca098da92be7c1e0592 (127.28.25.3:46095): Not found: Tablet not found: 39f5dc8ca16f48889b3d792bc65cd23b
I20260811 11:19:48.010552 14917 leader_election.cc:302] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efbc2cdc45804ab3b5ba931af9f808f7; no voters: b1a9983e15044ca098da92be7c1e0592, deefe38bc52947db99da0880d75201f7
I20260811 11:19:48.010634 15376 raft_consensus.cc:2726] T 39f5dc8ca16f48889b3d792bc65cd23b P efbc2cdc45804ab3b5ba931af9f808f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:19:48.717691 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:48.717342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:44133)
0811 11:19:48.717592 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:48.717641 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":135,"threads_started":1}
I20260811 11:19:48.767812 15379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:48.767536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:37199)
0811 11:19:48.767705 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:48.767754 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260811 11:19:48.907133 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:48.906836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:51067)
0811 11:19:48.907023 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:48.907081 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260811 11:19:49.718736 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:49.718386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:41397)
0811 11:19:49.718627 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:49.718673 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260811 11:19:49.768610 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:49.768349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:48119)
0811 11:19:49.768514 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:49.768560 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260811 11:19:49.907945 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:49.907678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:59493)
0811 11:19:49.907845 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:49.907891 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260811 11:19:50.719470 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:50.719224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:58293)
0811 11:19:50.719384 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:50.719420 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260811 11:19:50.769459 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:50.769170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:42357)
0811 11:19:50.769352 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:50.769403 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260811 11:19:50.908782 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:50.908484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:38663)
0811 11:19:50.908650 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:50.908699 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260811 11:19:51.720368 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:51.720112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:52207)
0811 11:19:51.720272 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:51.720315 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
I20260811 11:19:51.770416 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:51.770114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:55443)
0811 11:19:51.770299 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:51.770348 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260811 11:19:51.909869 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:51.909552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:39067)
0811 11:19:51.909768 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:51.909816 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260811 11:19:52.721211 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:52.720944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:47679)
0811 11:19:52.721111 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:52.721159 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260811 11:19:52.771432 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:52.771178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:44319)
0811 11:19:52.771329 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:52.771382 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260811 11:19:52.910869 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:52.910518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:45377)
0811 11:19:52.910741 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:52.910802 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260811 11:19:53.722223 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:53.721942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.1:39113)
0811 11:19:53.722114 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:53.722161 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260811 11:19:53.772296 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:53.772041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.2:55247)
0811 11:19:53.772199 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:53.772245 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260811 11:19:53.912000 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 14860
I20260811 11:19:53.913789 15396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:53.911700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33593 (local address 127.28.25.3:40739)
0811 11:19:53.911949 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:53.912110 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.62:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
I20260811 11:19:53.920591 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 15044
I20260811 11:19:53.929141 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 15194
2026-08-11T11:19:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59405 ms)
[----------] 30 tests from SecurityITest (282835 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:19:54 dist-test-slave-2w3w krb5kdc[15399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:19:54 dist-test-slave-2w3w krb5kdc[15399](info): set up 2 sockets
Aug 11 11:19:54 dist-test-slave-2w3w krb5kdc[15399](info): commencing operation
krb5kdc: starting...
W20260811 11:19:55.986107 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:19:56 dist-test-slave-2w3w krb5kdc[15399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:19:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:19:56.139377 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:46121
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:46121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:19:56.229158 15413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:56.229379 15413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:56.229403 15413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:56.230602 15413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:19:56.230647 15413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:56.230662 15413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:56.230675 15413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:19:56.230687 15413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:19:56.232343 15413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:46121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:56.232627 15413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:56.232882 15413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:56.238533 15422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:56.238543 15419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:56.238600 15413 server_base.cc:1034] running on GCE node
W20260811 11:19:56.238548 15420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:56.239079 15413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:56.239410 15413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:56.240564 15413 hybrid_clock.cc:648] HybridClock initialized: now 1786447196240549 us; error 31 us; skew 500 ppm
Aug 11 11:19:56 dist-test-slave-2w3w krb5kdc[15399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:56.243238 15413 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:19:56.243741 15413 webserver.cc:460] Webserver started at http://127.28.25.62:44827/ using document root <none> and password file <none>
I20260811 11:19:56.243984 15413 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:56.244055 15413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:56.244210 15413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:56.245362 15413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "6c34a8e93150403b868e8e89fb7fb5c8"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "cf87c9d30f510524f385ee375ee08838"
server_key_iv: "fde994e5f487779152cf2d8791e95263"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.245741 15413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "6c34a8e93150403b868e8e89fb7fb5c8"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "cf87c9d30f510524f385ee375ee08838"
server_key_iv: "fde994e5f487779152cf2d8791e95263"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.247330 15413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260811 11:19:56.248219 15432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:56.248627 15413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:56.248765 15413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "6c34a8e93150403b868e8e89fb7fb5c8"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "cf87c9d30f510524f385ee375ee08838"
server_key_iv: "fde994e5f487779152cf2d8791e95263"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.248876 15413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:56.262797 15413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:56.265219 15413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:56.265437 15413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:56.270130 15413 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:46121
I20260811 11:19:56.270166 15488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:46121 every 8 connection(s)
I20260811 11:19:56.270566 15413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:19:56.271570 15489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:56.273886 15489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8: Bootstrap starting.
I20260811 11:19:56.274479 15489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:56.274749 15489 log.cc:824] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:56.275559 15489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8: No bootstrap required, opened a new log
I20260811 11:19:56.275941 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 15413
I20260811 11:19:56.276073 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:19:56.276295 28772 external_mini_cluster.cc:1428] Setting key e5ade3f9257b2f0ed9afc41d74caa212
I20260811 11:19:56.277635 15489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c34a8e93150403b868e8e89fb7fb5c8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46121 } }
I20260811 11:19:56.277819 15489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:56.277885 15489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c34a8e93150403b868e8e89fb7fb5c8, State: Initialized, Role: FOLLOWER
I20260811 11:19:56.278045 15489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c34a8e93150403b868e8e89fb7fb5c8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46121 } }
I20260811 11:19:56.278141 15489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:19:56.278178 15489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:19:56.278230 15489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:19:56.279390 15489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c34a8e93150403b868e8e89fb7fb5c8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46121 } }
I20260811 11:19:56.279532 15489 leader_election.cc:302] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c34a8e93150403b868e8e89fb7fb5c8; no voters:
I20260811 11:19:56.279749 15489 leader_election.cc:288] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:19:56.279767 15494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:19:56.279891 15494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [term 1 LEADER]: Becoming Leader. State: Replica: 6c34a8e93150403b868e8e89fb7fb5c8, State: Running, Role: LEADER
I20260811 11:19:56.280048 15489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:19:56.280041 15494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c34a8e93150403b868e8e89fb7fb5c8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46121 } }
I20260811 11:19:56.280685 15496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c34a8e93150403b868e8e89fb7fb5c8. Latest consensus state: current_term: 1 leader_uuid: "6c34a8e93150403b868e8e89fb7fb5c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c34a8e93150403b868e8e89fb7fb5c8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46121 } } }
I20260811 11:19:56.280781 15496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [sys.catalog]: This master's current role is: LEADER
I20260811 11:19:56.281330 15495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c34a8e93150403b868e8e89fb7fb5c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c34a8e93150403b868e8e89fb7fb5c8" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46121 } } }
I20260811 11:19:56.281420 15495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8 [sys.catalog]: This master's current role is: LEADER
I20260811 11:19:56.282063 15503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:19:56.282626 15503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 11 11:19:56 dist-test-slave-2w3w krb5kdc[15399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:19:56.284718 15503 catalog_manager.cc:1347] Generated new cluster ID: 4ddae1fd5420445bab99a5356b4a9f03
I20260811 11:19:56.284781 15503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:19:56.289285 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.276910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.28.25.62:46121)
0811 11:19:56.277090 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:56.277098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:56.277126 (+ 28us) server_negotiation.cc:407] Connection header received
0811 11:19:56.277623 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:56.277631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:56.277688 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:56.277831 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:56.278734 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.279997 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.280969 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.281918 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.282428 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.285240 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:56.285245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:56.285245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:56.285261 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:56.287446 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:56.287885 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:56.287889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:56.287889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:56.288018 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:56.288282 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:56.288283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:56.288284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:56.288569 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:56.288769 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:56.288874 (+ 105us) server_negotiation.cc:299] Negotiation successful
0811 11:19:56.289050 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260811 11:19:56.307715 15503 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:19:56.308349 15503 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:19:56.314040 15503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8: Generated new TSK 0
I20260811 11:19:56.314325 15503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:19:56.346812 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46121
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:19:56.443609 15516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:56.443856 15516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:56.443886 15516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:56.445178 15516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:56.445220 15516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:56.445251 15516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:19:56.446916 15516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:56.447233 15516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:56.447461 15516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:56.453218 15522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:56.453253 15521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:56.453218 15525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:56.453608 15516 server_base.cc:1034] running on GCE node
I20260811 11:19:56.453859 15516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:56.454145 15516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:56.455386 15516 hybrid_clock.cc:648] HybridClock initialized: now 1786447196455384 us; error 63 us; skew 500 ppm
Aug 11 11:19:56 dist-test-slave-2w3w krb5kdc[15399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:56.458323 15516 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:19:56.458783 15516 webserver.cc:460] Webserver started at http://127.28.25.1:45357/ using document root <none> and password file <none>
I20260811 11:19:56.459065 15516 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:56.459118 15516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:56.459295 15516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:56.460403 15516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "612b8fe2a5174cba956d40ad8dabbcec"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "23817317ddc069fe6718725fcd5177f5"
server_key_iv: "c1a4b8a4552bb868b4d6d5a0829db4de"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.460800 15516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "612b8fe2a5174cba956d40ad8dabbcec"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "23817317ddc069fe6718725fcd5177f5"
server_key_iv: "c1a4b8a4552bb868b4d6d5a0829db4de"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.462316 15516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:19:56.463173 15536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:56.463430 15516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:19:56.463514 15516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "612b8fe2a5174cba956d40ad8dabbcec"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "23817317ddc069fe6718725fcd5177f5"
server_key_iv: "c1a4b8a4552bb868b4d6d5a0829db4de"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.463614 15516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:56.478114 15516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:56.480932 15516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:56.481114 15516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:56.481462 15516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:56.481884 15516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:56.481923 15516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:56.481972 15516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:56.482002 15516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:56.489244 15516 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:37541
I20260811 11:19:56.489331 15657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:37541 every 8 connection(s)
I20260811 11:19:56.489735 15516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:19:56.493510 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 15516
I20260811 11:19:56.493649 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:19:56.493872 28772 external_mini_cluster.cc:1428] Setting key 09ab593df7ea43d44d325875e77b5ddf
Aug 11 11:19:56 dist-test-slave-2w3w krb5kdc[15399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:19:56.501448 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.490633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:47921 (local address 127.28.25.62:46121)
0811 11:19:56.490706 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:56.490708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:56.490998 (+ 290us) server_negotiation.cc:407] Connection header received
0811 11:19:56.491603 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:56.491604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:56.491621 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:56.491683 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:56.492866 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.493408 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.494797 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.495110 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.495198 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.497950 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:56.497952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:56.497952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:56.497964 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:56.499594 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:56.500201 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:56.500202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:56.500202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:56.500264 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:56.500642 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:56.500644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:56.500644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:56.500787 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:56.500855 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:56.501268 (+ 413us) server_negotiation.cc:299] Negotiation successful
0811 11:19:56.501373 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:19:56.501752 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.490613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46121 (local address 127.28.25.1:47921)
0811 11:19:56.490833 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:56.490874 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:56.491481 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:56.491770 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:56.491773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:56.492545 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:56.492762 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.492767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.493531 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.493532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.494680 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.494683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.495212 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.495213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.495360 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.496014 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:19:56.496023 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:56.497835 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:56.499692 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:56.499702 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:56.499705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:56.500077 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:56.500360 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:56.500361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:56.500361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:56.500530 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:56.500876 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:56.500885 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:56.501096 (+ 211us) client_negotiation.cc:769] Sending connection context
0811 11:19:56.501239 (+ 143us) client_negotiation.cc:240] Negotiation successful
0811 11:19:56.501455 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":115,"threads_started":1}
I20260811 11:19:56.502463 15658 heartbeater.cc:347] Connected to a master server at 127.28.25.62:46121
I20260811 11:19:56.502576 15658 heartbeater.cc:464] Registering TS with master...
I20260811 11:19:56.502799 15658 heartbeater.cc:511] Master 127.28.25.62:46121 requested a full tablet report, sending...
I20260811 11:19:56.503417 15452 ts_manager.cc:194] Registered new tserver with Master: 612b8fe2a5174cba956d40ad8dabbcec (127.28.25.1:37541)
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
I20260811 11:19:56.504083 15452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:47921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:19:56.527967 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46121
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:19:56.616273 15663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:56.616516 15663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:56.616544 15663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:56.617734 15663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:56.617779 15663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:56.617805 15663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:19:56.619356 15663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:56.619582 15663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:56.619787 15663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:56.625298 15673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:56.625319 15670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:56.625361 15671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:56.625526 15663 server_base.cc:1034] running on GCE node
I20260811 11:19:56.625922 15663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:56.626192 15663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:56.627363 15663 hybrid_clock.cc:648] HybridClock initialized: now 1786447196627353 us; error 34 us; skew 500 ppm
Aug 11 11:19:56 dist-test-slave-2w3w krb5kdc[15399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:56.630350 15663 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:19:56.630913 15663 webserver.cc:460] Webserver started at http://127.28.25.2:39935/ using document root <none> and password file <none>
I20260811 11:19:56.631229 15663 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:56.631309 15663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:56.631479 15663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:56.632661 15663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "4f6c06341687f3ae6e41ac574ce2e801"
server_key_iv: "1a5d38c17ab3563f2f52049e08e10915"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.633067 15663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "4f6c06341687f3ae6e41ac574ce2e801"
server_key_iv: "1a5d38c17ab3563f2f52049e08e10915"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.634756 15663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:19:56.635869 15680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:56.636163 15663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:56.636328 15663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "4f6c06341687f3ae6e41ac574ce2e801"
server_key_iv: "1a5d38c17ab3563f2f52049e08e10915"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.636426 15663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:56.660746 15663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:56.663095 15663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:56.663272 15663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:56.663544 15663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:56.663878 15663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:56.663944 15663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:56.663988 15663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:56.664029 15663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:56.670712 15663 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:42401
I20260811 11:19:56.670763 15802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:42401 every 8 connection(s)
I20260811 11:19:56.671207 15663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:19:56.675304 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 15663
I20260811 11:19:56.675437 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:19:56.675665 28772 external_mini_cluster.cc:1428] Setting key 65462c1e3cadd984446b867d66c8c22b
Aug 11 11:19:56 dist-test-slave-2w3w krb5kdc[15399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:19:56.684253 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.672116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:46835 (local address 127.28.25.62:46121)
0811 11:19:56.672282 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:56.672284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:56.672438 (+ 154us) server_negotiation.cc:407] Connection header received
0811 11:19:56.672957 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:56.673101 (+ 144us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:56.673126 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:56.673210 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:56.674387 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.675069 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.676566 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.676813 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.676885 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.680213 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:56.680215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:56.680216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:56.680226 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:56.682123 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:56.682879 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:56.682880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:56.682880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:56.682948 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:56.683396 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:56.683397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:56.683397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:56.683541 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:56.683607 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:56.684070 (+ 463us) server_negotiation.cc:299] Negotiation successful
0811 11:19:56.684173 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260811 11:19:56.684557 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.672090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46121 (local address 127.28.25.2:46835)
0811 11:19:56.672287 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:56.672334 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:56.672853 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:56.673334 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:56.673338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:56.674045 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:56.674264 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.674270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.675178 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.675180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.676481 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.676485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.676920 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.676921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.677083 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.677684 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:19:56.677695 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:56.680055 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:56.682248 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:56.682261 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:56.682264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:56.682720 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:56.683102 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:56.683103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:56.683104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:56.683295 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:56.683626 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:56.683635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:56.683874 (+ 239us) client_negotiation.cc:769] Sending connection context
0811 11:19:56.684027 (+ 153us) client_negotiation.cc:240] Negotiation successful
0811 11:19:56.684265 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":117,"threads_started":1}
I20260811 11:19:56.685191 15803 heartbeater.cc:347] Connected to a master server at 127.28.25.62:46121
I20260811 11:19:56.685350 15803 heartbeater.cc:464] Registering TS with master...
I20260811 11:19:56.685572 15803 heartbeater.cc:511] Master 127.28.25.62:46121 requested a full tablet report, sending...
I20260811 11:19:56.686102 15452 ts_manager.cc:194] Registered new tserver with Master: 48c0f418c5f54dfa8aec7b3c40bebbaa (127.28.25.2:42401)
I20260811 11:19:56.686519 15452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:46835
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:19:56.708348 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46121
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:19:56.797999 15808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:56.798236 15808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:56.798265 15808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:56.799646 15808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:56.799698 15808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:56.799728 15808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:19:56.801397 15808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:56.801681 15808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:56.801965 15808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:56.807559 15818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:56.807606 15815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:56.807802 15808 server_base.cc:1034] running on GCE node
W20260811 11:19:56.807816 15816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:56.808075 15808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:56.808364 15808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:56.809557 15808 hybrid_clock.cc:648] HybridClock initialized: now 1786447196809536 us; error 59 us; skew 500 ppm
Aug 11 11:19:56 dist-test-slave-2w3w krb5kdc[15399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:56.812562 15808 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:19:56.813127 15808 webserver.cc:460] Webserver started at http://127.28.25.3:33519/ using document root <none> and password file <none>
I20260811 11:19:56.813426 15808 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:56.813503 15808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:56.813730 15808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:56.815056 15808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "ee05ce48f88d40ce81574b27df3a2dce"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "ae93162a2c07c269f42058d6bbad89d0"
server_key_iv: "ac541093aed2dfcec040391ab1336f5c"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.815443 15808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "ee05ce48f88d40ce81574b27df3a2dce"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "ae93162a2c07c269f42058d6bbad89d0"
server_key_iv: "ac541093aed2dfcec040391ab1336f5c"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.816890 15808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:19:56.817797 15826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:56.818080 15808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:56.818204 15808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "ee05ce48f88d40ce81574b27df3a2dce"
format_stamp: "Formatted at 2026-08-11 11:19:56 on dist-test-slave-2w3w"
server_key: "ae93162a2c07c269f42058d6bbad89d0"
server_key_iv: "ac541093aed2dfcec040391ab1336f5c"
server_key_version: "encryptionkey@0"
I20260811 11:19:56.818320 15808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:56.852049 15808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:56.854694 15808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:56.854897 15808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:56.855223 15808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:56.855566 15808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:56.855624 15808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:56.855669 15808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:56.855695 15808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:56.862219 15808 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:38879
I20260811 11:19:56.862349 15951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:38879 every 8 connection(s)
I20260811 11:19:56.862663 15808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:19:56.865352 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 15808
I20260811 11:19:56.865453 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:19:56.865654 28772 external_mini_cluster.cc:1428] Setting key 84b93c00062de843de0a72fc9187a3fa
Aug 11 11:19:56 dist-test-slave-2w3w krb5kdc[15399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:19:56.873647 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.863582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:47531 (local address 127.28.25.62:46121)
0811 11:19:56.863647 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:56.863649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:56.863937 (+ 288us) server_negotiation.cc:407] Connection header received
0811 11:19:56.864502 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:56.864504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:56.864518 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:56.864570 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:56.865615 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.866370 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.867686 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.867911 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.867989 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.870594 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:56.870596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:56.870597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:56.870606 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:56.872122 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:56.872596 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:56.872598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:56.872598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:56.872664 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:56.872939 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:56.872940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:56.872940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:56.873067 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:56.873149 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:56.873454 (+ 305us) server_negotiation.cc:299] Negotiation successful
0811 11:19:56.873544 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 11:19:56.873880 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.863575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:46121 (local address 127.28.25.3:47531)
0811 11:19:56.863735 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:56.863841 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:56.864381 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:56.864646 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:56.864649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:56.865246 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:56.865506 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.865512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.866449 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.866451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.867585 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.867589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.868010 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.868011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.868140 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.868732 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:19:56.868741 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:56.870477 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:56.872237 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:56.872247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:56.872249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:56.872509 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:56.872744 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:56.872745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:56.872745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:56.872868 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:56.873141 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:56.873148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:56.873312 (+ 164us) client_negotiation.cc:769] Sending connection context
0811 11:19:56.873430 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 11:19:56.873617 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260811 11:19:56.874648 15952 heartbeater.cc:347] Connected to a master server at 127.28.25.62:46121
I20260811 11:19:56.874781 15952 heartbeater.cc:464] Registering TS with master...
I20260811 11:19:56.875098 15952 heartbeater.cc:511] Master 127.28.25.62:46121 requested a full tablet report, sending...
I20260811 11:19:56.875525 15452 ts_manager.cc:194] Registered new tserver with Master: ee05ce48f88d40ce81574b27df3a2dce (127.28.25.3:38879)
I20260811 11:19:56.876020 15452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:47531
I20260811 11:19:56.877326 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:19:56.885339 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.878583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.28.25.62:46121)
0811 11:19:56.878657 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:56.878660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:56.878763 (+ 103us) server_negotiation.cc:407] Connection header received
0811 11:19:56.878820 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:56.878821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:56.878831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:56.878878 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:56.879518 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.880021 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.880997 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.881287 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.881383 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.882655 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:56.882657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:56.882657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:56.882663 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:56.884190 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:56.884571 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:56.884572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:56.884572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:56.884623 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:56.884816 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:56.884816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:56.884817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:56.885007 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:56.885085 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:56.885210 (+ 125us) server_negotiation.cc:299] Negotiation successful
0811 11:19:56.885269 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:19:56.887215 15452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:19:56.888921 15452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:19:56.896140 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.891493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:37541 (local address 127.0.0.1:36232)
0811 11:19:56.891956 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:56.891964 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:56.892000 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:56.892639 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:56.892640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:56.892642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:56.892685 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.892686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.893929 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.893930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.895041 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.895043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.895757 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.895758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.895853 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.895869 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 11:19:56.895974 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 11:19:56.896046 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":121,"threads_started":1}
I20260811 11:19:56.896404 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.891483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:42401 (local address 127.0.0.1:53486)
0811 11:19:56.891671 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:56.891713 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:56.891785 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:56.892343 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:56.892346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:56.892349 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:56.892432 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.892435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.893585 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.893586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.894858 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.894861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.895973 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.895974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.896031 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.896034 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:19:56.896089 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 11:19:56.896123 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":119,"threads_started":1}
I20260811 11:19:56.896241 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.891680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.28.25.2:42401)
0811 11:19:56.892130 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:56.892136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:56.892149 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:19:56.892201 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:56.892204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:56.892222 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:56.892307 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:56.892499 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.893479 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.894959 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.895879 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.895968 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.896027 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:56.896103 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 11:19:56.896147 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":108,"threads_started":1}
I20260811 11:19:56.896565 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.891767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.28.25.1:37541)
0811 11:19:56.892478 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:56.892482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:56.892492 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:19:56.892530 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:56.892532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:56.892548 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:56.892630 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:56.892749 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.893821 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.895127 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.895654 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.895754 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.895823 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:56.896199 (+ 376us) server_negotiation.cc:299] Negotiation successful
0811 11:19:56.896377 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":122,"threads_started":1}
I20260811 11:19:56.897439 15587 tablet_service.cc:1467] Processing CreateTablet for tablet 80018f010fe4469089b41c666e88f2a0 (DEFAULT_TABLE table=test-table [id=c77d76dce87946b1932fcf3daa06bc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:19:56.897439 15730 tablet_service.cc:1467] Processing CreateTablet for tablet 80018f010fe4469089b41c666e88f2a0 (DEFAULT_TABLE table=test-table [id=c77d76dce87946b1932fcf3daa06bc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:19:56.897787 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.891483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:38879 (local address 127.0.0.1:56586)
0811 11:19:56.891768 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:56.891781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:56.891841 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:56.892380 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:56.892381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:56.892384 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:56.892442 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.892444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.895048 (+ 2604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.895049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.896298 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:56.896302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.897501 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.897502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:56.897576 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.897582 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:19:56.897654 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 11:19:56.897703 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260811 11:19:56.897804 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:56.891912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.28.25.3:38879)
0811 11:19:56.892196 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:56.892200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:56.892208 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:19:56.892255 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:56.892257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:56.892272 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:56.892526 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:56.892614 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.893806 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.896410 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:56.897380 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:56.897500 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:56.897568 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:56.897671 (+ 103us) server_negotiation.cc:299] Negotiation successful
0811 11:19:56.897716 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260811 11:19:56.897833 15587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80018f010fe4469089b41c666e88f2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:56.897833 15730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80018f010fe4469089b41c666e88f2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:56.898489 15880 tablet_service.cc:1467] Processing CreateTablet for tablet 80018f010fe4469089b41c666e88f2a0 (DEFAULT_TABLE table=test-table [id=c77d76dce87946b1932fcf3daa06bc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:19:56.898752 15880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80018f010fe4469089b41c666e88f2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:56.901422 15970 tablet_bootstrap.cc:492] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa: Bootstrap starting.
I20260811 11:19:56.901715 15971 tablet_bootstrap.cc:492] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec: Bootstrap starting.
I20260811 11:19:56.901954 15972 tablet_bootstrap.cc:492] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: Bootstrap starting.
I20260811 11:19:56.902617 15972 tablet_bootstrap.cc:654] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:56.902686 15971 tablet_bootstrap.cc:654] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:56.903245 15972 log.cc:824] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:56.903424 15971 log.cc:824] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:56.903820 15970 tablet_bootstrap.cc:654] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:56.904511 15971 tablet_bootstrap.cc:492] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec: No bootstrap required, opened a new log
I20260811 11:19:56.904506 15970 log.cc:824] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:56.904579 15971 ts_tablet_manager.cc:1397] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:19:56.905541 15970 tablet_bootstrap.cc:492] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa: No bootstrap required, opened a new log
I20260811 11:19:56.905752 15970 ts_tablet_manager.cc:1397] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 11:19:56.906953 15971 raft_consensus.cc:348] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 } } peers { permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 } } peers { permanent_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42401 } }
I20260811 11:19:56.907226 15971 raft_consensus.cc:374] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:56.907279 15971 raft_consensus.cc:729] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 612b8fe2a5174cba956d40ad8dabbcec, State: Initialized, Role: FOLLOWER
I20260811 11:19:56.907433 15971 consensus_queue.cc:260] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 } } peers { permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 } } peers { permanent_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42401 } }
I20260811 11:19:56.907554 15972 tablet_bootstrap.cc:492] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: No bootstrap required, opened a new log
I20260811 11:19:56.907625 15972 ts_tablet_manager.cc:1397] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260811 11:19:56.907757 15971 ts_tablet_manager.cc:1428] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260811 11:19:56.907927 15658 heartbeater.cc:503] Master 127.28.25.62:46121 was elected leader, sending a full tablet report...
I20260811 11:19:56.908144 15970 raft_consensus.cc:348] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 } } peers { permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 } } peers { permanent_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42401 } }
I20260811 11:19:56.908401 15970 raft_consensus.cc:374] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:56.908460 15970 raft_consensus.cc:729] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c0f418c5f54dfa8aec7b3c40bebbaa, State: Initialized, Role: FOLLOWER
I20260811 11:19:56.908587 15970 consensus_queue.cc:260] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 } } peers { permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 } } peers { permanent_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42401 } }
I20260811 11:19:56.908957 15970 ts_tablet_manager.cc:1428] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:19:56.909116 15803 heartbeater.cc:503] Master 127.28.25.62:46121 was elected leader, sending a full tablet report...
I20260811 11:19:56.909885 15972 raft_consensus.cc:348] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 } } peers { permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 } } peers { permanent_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42401 } }
I20260811 11:19:56.910059 15972 raft_consensus.cc:374] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:56.910106 15972 raft_consensus.cc:729] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee05ce48f88d40ce81574b27df3a2dce, State: Initialized, Role: FOLLOWER
I20260811 11:19:56.910240 15972 consensus_queue.cc:260] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 } } peers { permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 } } peers { permanent_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42401 } }
I20260811 11:19:56.910490 15972 ts_tablet_manager.cc:1428] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 11:19:56.910574 15952 heartbeater.cc:503] Master 127.28.25.62:46121 was elected leader, sending a full tablet report...
W20260811 11:19:56.921976 15804 tablet.cc:2369] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:19:56.990640 15659 tablet.cc:2369] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:19:57.113384 15953 tablet.cc:2369] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:19:57.176558 15977 raft_consensus.cc:482] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:19:57.176709 15977 raft_consensus.cc:504] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 } } peers { permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 } } peers { permanent_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42401 } }
I20260811 11:19:57.177299 15977 leader_election.cc:288] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee05ce48f88d40ce81574b27df3a2dce (127.28.25.3:38879), 612b8fe2a5174cba956d40ad8dabbcec (127.28.25.1:37541)
I20260811 11:19:57.180733 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.177774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:38393 (local address 127.28.25.3:38879)
0811 11:19:57.177895 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.177897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.177929 (+ 32us) server_negotiation.cc:407] Connection header received
0811 11:19:57.177959 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.177960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.177973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.178053 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:57.178222 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.178773 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.179915 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.180370 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.180454 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.180465 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.180626 (+ 161us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.180665 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 11:19:57.180742 15979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.177620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:38879 (local address 127.28.25.2:38393)
0811 11:19:57.177788 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:57.177799 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:57.177848 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:57.178058 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:57.178059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:57.178061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:57.178148 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.178149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.178871 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.178873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.179809 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.179813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.180456 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.180457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.180539 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.180543 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:19:57.180606 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 11:19:57.180651 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260811 11:19:57.180819 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.177577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:59235 (local address 127.28.25.1:37541)
0811 11:19:57.177648 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.177651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.177669 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:19:57.177707 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.177708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.177725 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.177796 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:57.178041 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.178632 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.179905 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.180419 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.180503 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.180513 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.180690 (+ 177us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.180731 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 11:19:57.181134 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.177515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:37541 (local address 127.28.25.2:59235)
0811 11:19:57.177581 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:57.177594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:57.177640 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:57.177865 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:57.177866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:57.177867 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:57.177966 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.177968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.178735 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.178736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.179793 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.179796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.180503 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.180504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.180598 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.180602 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:19:57.180669 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 11:19:57.180716 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260811 11:19:57.181309 15901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80018f010fe4469089b41c666e88f2a0" candidate_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee05ce48f88d40ce81574b27df3a2dce" is_pre_election: true
I20260811 11:19:57.181522 15901 raft_consensus.cc:2454] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c0f418c5f54dfa8aec7b3c40bebbaa in term 0.
I20260811 11:19:57.181552 15609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80018f010fe4469089b41c666e88f2a0" candidate_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "612b8fe2a5174cba956d40ad8dabbcec" is_pre_election: true
I20260811 11:19:57.181747 15609 raft_consensus.cc:2454] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c0f418c5f54dfa8aec7b3c40bebbaa in term 0.
I20260811 11:19:57.181842 15681 leader_election.cc:302] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c0f418c5f54dfa8aec7b3c40bebbaa, ee05ce48f88d40ce81574b27df3a2dce; no voters:
I20260811 11:19:57.182003 15977 raft_consensus.cc:2781] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:19:57.182065 15977 raft_consensus.cc:482] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:19:57.182090 15977 raft_consensus.cc:3037] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:19:57.182873 15977 raft_consensus.cc:504] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 } } peers { permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 } } peers { permanent_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42401 } }
I20260811 11:19:57.183070 15977 leader_election.cc:288] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [CANDIDATE]: Term 1 election: Requested vote from peers ee05ce48f88d40ce81574b27df3a2dce (127.28.25.3:38879), 612b8fe2a5174cba956d40ad8dabbcec (127.28.25.1:37541)
I20260811 11:19:57.183226 15901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80018f010fe4469089b41c666e88f2a0" candidate_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee05ce48f88d40ce81574b27df3a2dce"
I20260811 11:19:57.183266 15609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80018f010fe4469089b41c666e88f2a0" candidate_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "612b8fe2a5174cba956d40ad8dabbcec"
I20260811 11:19:57.183311 15901 raft_consensus.cc:3037] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:19:57.183359 15609 raft_consensus.cc:3037] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:19:57.184006 15901 raft_consensus.cc:2454] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c0f418c5f54dfa8aec7b3c40bebbaa in term 1.
I20260811 11:19:57.184198 15681 leader_election.cc:302] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c0f418c5f54dfa8aec7b3c40bebbaa, ee05ce48f88d40ce81574b27df3a2dce; no voters:
I20260811 11:19:57.184266 15609 raft_consensus.cc:2454] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c0f418c5f54dfa8aec7b3c40bebbaa in term 1.
I20260811 11:19:57.184319 15977 raft_consensus.cc:2781] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:19:57.184496 15977 raft_consensus.cc:686] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 1 LEADER]: Becoming Leader. State: Replica: 48c0f418c5f54dfa8aec7b3c40bebbaa, State: Running, Role: LEADER
I20260811 11:19:57.184623 15977 consensus_queue.cc:237] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 } } peers { permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 } } peers { permanent_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42401 } }
I20260811 11:19:57.185510 15450 catalog_manager.cc:5495] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 48c0f418c5f54dfa8aec7b3c40bebbaa (127.28.25.2). New cstate: current_term: 1 leader_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c0f418c5f54dfa8aec7b3c40bebbaa" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42401 } health_report { overall_health: HEALTHY } } }
I20260811 11:19:57.254873 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.252029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.28.25.2:42401)
0811 11:19:57.252105 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.252107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.252116 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:19:57.252159 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.252160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.252182 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.252249 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:19:57.252460 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.253016 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.253932 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.254144 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.254207 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.254467 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:19:57.254584 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:19:57.254663 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.254701 (+ 38us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.254738 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:19:57.257045 15609 raft_consensus.cc:1261] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec [term 1 FOLLOWER]: Refusing update from remote peer 48c0f418c5f54dfa8aec7b3c40bebbaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:19:57.257045 15901 raft_consensus.cc:1261] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce [term 1 FOLLOWER]: Refusing update from remote peer 48c0f418c5f54dfa8aec7b3c40bebbaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:19:57.257431 15977 consensus_queue.cc:1035] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee05ce48f88d40ce81574b27df3a2dce" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:19:57.257588 15980 consensus_queue.cc:1035] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "612b8fe2a5174cba956d40ad8dabbcec" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:19:57.260553 15985 mvcc.cc:205] Tried to move back new op lower bound from 7317287719961583616 to 7317287719668760576. Current Snapshot: MvccSnapshot[applied={T|T < 7317287719961583616}]
I20260811 11:19:57.261279 15984 mvcc.cc:205] Tried to move back new op lower bound from 7317287719961583616 to 7317287719668760576. Current Snapshot: MvccSnapshot[applied={T|T < 7317287719961583616 or (T in {7317287719961583616})}]
I20260811 11:19:57.262809 15450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:19:57.262908 15450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:19:57.263713 15450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c34a8e93150403b868e8e89fb7fb5c8: Sending DeleteTablet for 3 replicas of tablet 80018f010fe4469089b41c666e88f2a0
I20260811 11:19:57.264076 15880 tablet_service.cc:1514] Processing DeleteTablet for tablet 80018f010fe4469089b41c666e88f2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:19:57 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:56586
I20260811 11:19:57.264134 15587 tablet_service.cc:1514] Processing DeleteTablet for tablet 80018f010fe4469089b41c666e88f2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:19:57 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:36232
I20260811 11:19:57.264184 15730 tablet_service.cc:1514] Processing DeleteTablet for tablet 80018f010fe4469089b41c666e88f2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:19:57 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:53486
I20260811 11:19:57.264451 15995 tablet_replica.cc:330] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: stopping tablet replica
I20260811 11:19:57.264566 15995 raft_consensus.cc:2229] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:19:57.264667 15997 tablet_replica.cc:330] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa: stopping tablet replica
I20260811 11:19:57.264688 15995 raft_consensus.cc:2258] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:19:57.264678 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 15516
I20260811 11:19:57.266624 15995 ts_tablet_manager.cc:1905] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:19:57.269677 15995 ts_tablet_manager.cc:1918] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:19:57.269773 15995 log.cc:1196] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/80018f010fe4469089b41c666e88f2a0
I20260811 11:19:57.270041 15995 ts_tablet_manager.cc:1939] T 80018f010fe4469089b41c666e88f2a0 P ee05ce48f88d40ce81574b27df3a2dce: Deleting consensus metadata
I20260811 11:19:57.270658 15433 catalog_manager.cc:4841] TS ee05ce48f88d40ce81574b27df3a2dce (127.28.25.3:38879): tablet 80018f010fe4469089b41c666e88f2a0 (table test-table [id=c77d76dce87946b1932fcf3daa06bc9f]) successfully deleted
I20260811 11:19:57.275727 15997 raft_consensus.cc:2229] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:19:57.275918 15997 raft_consensus.cc:2258] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:19:57.264457 15996 tablet_replica.cc:330] T 80018f010fe4469089b41c666e88f2a0 P 612b8fe2a5174cba956d40ad8dabbcec: stopping tablet replica
W20260811 11:19:57.280755 15435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:37541 (error 108)
I20260811 11:19:57.281002 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 15663
I20260811 11:19:57.280982 15997 ts_tablet_manager.cc:1905] T 80018f010fe4469089b41c666e88f2a0 P 48c0f418c5f54dfa8aec7b3c40bebbaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:19:57.283551 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.283388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:37541 (local address 127.0.0.1:36244)
0811 11:19:57.283455 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:57.283512 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:37541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 11:19:57.283808 15435 catalog_manager.cc:4584] TS 612b8fe2a5174cba956d40ad8dabbcec (127.28.25.1:37541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80018f010fe4469089b41c666e88f2a0: Network error: Client connection negotiation failed: client connection to 127.28.25.1:37541: connect: Connection refused (error 111)
I20260811 11:19:57.288391 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 15808
I20260811 11:19:57.288435 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.288282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:42401 (local address 127.0.0.1:53498)
0811 11:19:57.288350 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:57.288389 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.2:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 11:19:57.296092 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 15413
2026-08-11T11:19:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2463 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-11T11:19:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:19:57Z Disabled control of system clock
I20260811 11:19:57.331591 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:41823
--webserver_interface=127.28.25.62
--webserver_port=0
--builtin_ntp_servers=127.28.25.20:38723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:41823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:19:57.424057 16003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:57.424343 16003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:57.424369 16003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:57.425717 16003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:19:57.425768 16003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:57.425783 16003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:57.425797 16003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:19:57.425810 16003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:19:57.427418 16003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:41823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:57.427664 16003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:57.427960 16003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:57.433516 16011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:57.433511 16009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:57.433522 16008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:57.433686 16003 server_base.cc:1034] running on GCE node
I20260811 11:19:57.434043 16003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:57.434369 16003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:57.435556 16003 hybrid_clock.cc:648] HybridClock initialized: now 1786447197435538 us; error 50 us; skew 500 ppm
I20260811 11:19:57.437099 16003 webserver.cc:460] Webserver started at http://127.28.25.62:35965/ using document root <none> and password file <none>
I20260811 11:19:57.437403 16003 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:57.437486 16003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:57.437646 16003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:57.438933 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "b59ed05832f84cdcb4dee7a07fddd32b"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "e18f43e9e3acd7db2e5d88f5def04c65"
server_key_iv: "efa4ea0c3bd8493c71d79563f1167809"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.439385 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "b59ed05832f84cdcb4dee7a07fddd32b"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "e18f43e9e3acd7db2e5d88f5def04c65"
server_key_iv: "efa4ea0c3bd8493c71d79563f1167809"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.440940 16003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:19:57.441879 16017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:57.442154 16003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:57.442281 16003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "b59ed05832f84cdcb4dee7a07fddd32b"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "e18f43e9e3acd7db2e5d88f5def04c65"
server_key_iv: "efa4ea0c3bd8493c71d79563f1167809"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.442386 16003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:57.463753 16003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:57.464160 16003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:57.464354 16003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:57.469708 16003 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:41823
I20260811 11:19:57.469807 16073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:41823 every 8 connection(s)
I20260811 11:19:57.470213 16003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:19:57.471400 16074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:57.473907 16074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b: Bootstrap starting.
I20260811 11:19:57.474511 16074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:57.474798 16074 log.cc:824] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:57.475582 16074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b: No bootstrap required, opened a new log
I20260811 11:19:57.477248 16074 raft_consensus.cc:348] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ed05832f84cdcb4dee7a07fddd32b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41823 } }
I20260811 11:19:57.477412 16074 raft_consensus.cc:374] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:57.477435 16074 raft_consensus.cc:729] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59ed05832f84cdcb4dee7a07fddd32b, State: Initialized, Role: FOLLOWER
I20260811 11:19:57.477587 16074 consensus_queue.cc:260] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ed05832f84cdcb4dee7a07fddd32b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41823 } }
I20260811 11:19:57.477671 16074 raft_consensus.cc:388] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:19:57.477694 16074 raft_consensus.cc:482] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:19:57.477723 16074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:19:57.478111 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 16003
I20260811 11:19:57.478227 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:19:57.478478 28772 external_mini_cluster.cc:1428] Setting key cba569c3c986fdf10477a2dff4da664f
I20260811 11:19:57.478509 16074 raft_consensus.cc:504] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ed05832f84cdcb4dee7a07fddd32b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41823 } }
I20260811 11:19:57.478617 16074 leader_election.cc:302] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b59ed05832f84cdcb4dee7a07fddd32b; no voters:
I20260811 11:19:57.478842 16074 leader_election.cc:288] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:19:57.479017 16077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:19:57.479178 16074 sys_catalog.cc:564] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:19:57.479223 16077 raft_consensus.cc:686] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [term 1 LEADER]: Becoming Leader. State: Replica: b59ed05832f84cdcb4dee7a07fddd32b, State: Running, Role: LEADER
I20260811 11:19:57.479348 16077 consensus_queue.cc:237] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ed05832f84cdcb4dee7a07fddd32b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41823 } }
I20260811 11:19:57.479739 16077 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b59ed05832f84cdcb4dee7a07fddd32b. Latest consensus state: current_term: 1 leader_uuid: "b59ed05832f84cdcb4dee7a07fddd32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ed05832f84cdcb4dee7a07fddd32b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41823 } } }
I20260811 11:19:57.479843 16077 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [sys.catalog]: This master's current role is: LEADER
I20260811 11:19:57.479869 16080 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b59ed05832f84cdcb4dee7a07fddd32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ed05832f84cdcb4dee7a07fddd32b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41823 } } }
I20260811 11:19:57.479976 16080 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b [sys.catalog]: This master's current role is: LEADER
W20260811 11:19:57.481195 16094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:19:57.481261 16094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:19:57.481307 16088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:19:57.481446 16088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:19:57.483268 16088 catalog_manager.cc:1347] Generated new cluster ID: bdbfc938f3154dd3bb28e8337e154489
I20260811 11:19:57.483328 16088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:19:57.485579 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.479158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.28.25.62:41823)
0811 11:19:57.479546 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.479556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.479596 (+ 40us) server_negotiation.cc:407] Connection header received
0811 11:19:57.479737 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.479746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.480066 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.480250 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:57.480571 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.482110 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.483210 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.483865 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.483987 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.484802 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:57.484806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:57.484806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:19:57.485021 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:57.485064 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:19:57.485078 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:57.485140 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.485246 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.485332 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260811 11:19:57.494997 16088 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:19:57.495572 16088 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:19:57.500298 16088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b: Generated new TSK 0
I20260811 11:19:57.500526 16088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:19:57.504381 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41823
--builtin_ntp_servers=127.28.25.20:38723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:19:57.595321 16099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:57.595608 16099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:57.595656 16099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:57.597157 16099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:57.597241 16099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:57.597294 16099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:19:57.599130 16099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:57.599525 16099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:57.599951 16099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:57.606585 16104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:57.606588 16107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:57.606575 16105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:57.607074 16099 server_base.cc:1034] running on GCE node
I20260811 11:19:57.607357 16099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:57.607700 16099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:57.608897 16099 hybrid_clock.cc:648] HybridClock initialized: now 1786447197608867 us; error 60 us; skew 500 ppm
I20260811 11:19:57.610585 16099 webserver.cc:460] Webserver started at http://127.28.25.1:45443/ using document root <none> and password file <none>
I20260811 11:19:57.610891 16099 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:57.610949 16099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:57.611178 16099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:57.612425 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "c7114d04379143b1b61d32964d6ac794"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "b2880a4b100ee5514ff5fb1f419fc6e5"
server_key_iv: "ee58f0c4673d8434a8d7d6f9150a2e84"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.612843 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "c7114d04379143b1b61d32964d6ac794"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "b2880a4b100ee5514ff5fb1f419fc6e5"
server_key_iv: "ee58f0c4673d8434a8d7d6f9150a2e84"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.614408 16099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:19:57.615399 16113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:57.615576 16099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:57.615694 16099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "c7114d04379143b1b61d32964d6ac794"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "b2880a4b100ee5514ff5fb1f419fc6e5"
server_key_iv: "ee58f0c4673d8434a8d7d6f9150a2e84"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.615783 16099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:57.634665 16099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:57.635159 16099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:57.635345 16099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:57.635628 16099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:57.636009 16099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:57.636049 16099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:57.636073 16099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:57.636087 16099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:57.643316 16099 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:33127
I20260811 11:19:57.643417 16233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:33127 every 8 connection(s)
I20260811 11:19:57.643775 16099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:19:57.649680 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.644704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:48693 (local address 127.28.25.62:41823)
0811 11:19:57.644840 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.644842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.645057 (+ 215us) server_negotiation.cc:407] Connection header received
0811 11:19:57.645688 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.645690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.645705 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.645763 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:57.646478 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.647103 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.648104 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.648322 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.648402 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.649380 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:57.649381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:57.649382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:19:57.649387 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:57.649402 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:19:57.649407 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:57.649451 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.649565 (+ 114us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.649597 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 11:19:57.649895 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.644660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:41823 (local address 127.28.25.1:48693)
0811 11:19:57.644930 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:57.644968 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:57.645539 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:57.645831 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:57.645835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:57.645845 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:57.646383 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.646388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.647201 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.647202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.648003 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.648005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.648412 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.648413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.648570 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.649224 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:19:57.649225 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:57.649259 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:19:57.649283 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:57.649462 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:57.649467 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:57.649471 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:19:57.649548 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 11:19:57.649653 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":117,"threads_started":1}
I20260811 11:19:57.650465 16234 heartbeater.cc:347] Connected to a master server at 127.28.25.62:41823
I20260811 11:19:57.650583 16234 heartbeater.cc:464] Registering TS with master...
I20260811 11:19:57.650772 16234 heartbeater.cc:511] Master 127.28.25.62:41823 requested a full tablet report, sending...
I20260811 11:19:57.651041 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 16099
I20260811 11:19:57.651170 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:19:57.651381 28772 external_mini_cluster.cc:1428] Setting key 98a220613a24cf7b65dfd1356bb5eccf
I20260811 11:19:57.651379 16036 ts_manager.cc:194] Registered new tserver with Master: c7114d04379143b1b61d32964d6ac794 (127.28.25.1:33127)
I20260811 11:19:57.652382 16036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.1:48693
I20260811 11:19:57.652839 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41823
--builtin_ntp_servers=127.28.25.20:38723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:19:57.748122 16239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:57.748382 16239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:57.748409 16239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:57.749778 16239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:57.749831 16239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:57.749862 16239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:19:57.751511 16239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:57.751821 16239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:57.752187 16239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:57.758175 16248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:57.758208 16246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:57.758173 16245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:57.758441 16239 server_base.cc:1034] running on GCE node
I20260811 11:19:57.758671 16239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:57.758903 16239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:57.760119 16239 hybrid_clock.cc:648] HybridClock initialized: now 1786447197760106 us; error 31 us; skew 500 ppm
I20260811 11:19:57.761543 16239 webserver.cc:460] Webserver started at http://127.28.25.2:38001/ using document root <none> and password file <none>
I20260811 11:19:57.761826 16239 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:57.761899 16239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:57.762060 16239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:57.763312 16239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "516786f85cae43e9b577482b58fcbebb"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "7bacc9ff79f39c2d4e3700358f4406ea"
server_key_iv: "3517814190ee1c13db0c0e080a1979d7"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.763746 16239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "516786f85cae43e9b577482b58fcbebb"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "7bacc9ff79f39c2d4e3700358f4406ea"
server_key_iv: "3517814190ee1c13db0c0e080a1979d7"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.765173 16239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:19:57.766002 16256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:57.766350 16239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:57.766494 16239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "516786f85cae43e9b577482b58fcbebb"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "7bacc9ff79f39c2d4e3700358f4406ea"
server_key_iv: "3517814190ee1c13db0c0e080a1979d7"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.766577 16239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:57.779179 16239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:57.779886 16239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:57.780474 16239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:57.780748 16239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:57.781132 16239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:57.781190 16239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:57.781235 16239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:57.781262 16239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:57.789271 16239 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:36223
I20260811 11:19:57.789321 16380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:36223 every 8 connection(s)
I20260811 11:19:57.789749 16239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:19:57.795718 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.790657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:33049 (local address 127.28.25.62:41823)
0811 11:19:57.790719 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.790722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.791062 (+ 340us) server_negotiation.cc:407] Connection header received
0811 11:19:57.791767 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.791768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.791800 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.791902 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:57.792636 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.793146 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.794185 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.794402 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.794469 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.795429 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:57.795431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:57.795431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:19:57.795436 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:57.795451 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:19:57.795455 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:57.795511 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.795623 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.795650 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 11:19:57.795929 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.790635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:41823 (local address 127.28.25.2:33049)
0811 11:19:57.790866 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:57.790917 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:57.791644 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:57.791973 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:57.791976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:57.791985 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:57.792529 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.792534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.793241 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.793242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.794085 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.794087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.794490 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.794491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.794604 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.795243 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:19:57.795244 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:57.795275 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:19:57.795300 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:57.795524 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:57.795529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:57.795532 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:19:57.795585 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 11:19:57.795678 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260811 11:19:57.796593 16382 heartbeater.cc:347] Connected to a master server at 127.28.25.62:41823
I20260811 11:19:57.796707 16382 heartbeater.cc:464] Registering TS with master...
I20260811 11:19:57.796926 16382 heartbeater.cc:511] Master 127.28.25.62:41823 requested a full tablet report, sending...
I20260811 11:19:57.797436 16036 ts_manager.cc:194] Registered new tserver with Master: 516786f85cae43e9b577482b58fcbebb (127.28.25.2:36223)
I20260811 11:19:57.797796 16036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.2:33049
I20260811 11:19:57.799171 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 16239
I20260811 11:19:57.799279 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:19:57.799504 28772 external_mini_cluster.cc:1428] Setting key 5186e3d553d9b607641d2a1fa56e2cc0
I20260811 11:19:57.800611 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41823
--builtin_ntp_servers=127.28.25.20:38723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:19:57.887708 16385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:57.887943 16385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:57.887969 16385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:57.889168 16385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:57.889213 16385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:57.889240 16385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:19:57.890647 16385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:57.890888 16385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:57.891276 16385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:57.896672 16391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:57.896678 16394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:57.896692 16390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:57.897104 16385 server_base.cc:1034] running on GCE node
I20260811 11:19:57.897351 16385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:57.897569 16385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:57.898726 16385 hybrid_clock.cc:648] HybridClock initialized: now 1786447197898722 us; error 69 us; skew 500 ppm
I20260811 11:19:57.900275 16385 webserver.cc:460] Webserver started at http://127.28.25.3:37255/ using document root <none> and password file <none>
I20260811 11:19:57.900565 16385 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:57.900619 16385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:57.900763 16385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:57.901943 16385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "8c7925f186e94d459e0ff46c6b624ac9"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "cb01105a7a4622c02b7a3e37a33949b6"
server_key_iv: "94e0cd5bf5aa0ee3fe88b9378f0654ef"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.902333 16385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "8c7925f186e94d459e0ff46c6b624ac9"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "cb01105a7a4622c02b7a3e37a33949b6"
server_key_iv: "94e0cd5bf5aa0ee3fe88b9378f0654ef"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.903903 16385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260811 11:19:57.904892 16400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:57.905107 16385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:57.905236 16385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "8c7925f186e94d459e0ff46c6b624ac9"
format_stamp: "Formatted at 2026-08-11 11:19:57 on dist-test-slave-2w3w"
server_key: "cb01105a7a4622c02b7a3e37a33949b6"
server_key_iv: "94e0cd5bf5aa0ee3fe88b9378f0654ef"
server_key_version: "encryptionkey@0"
I20260811 11:19:57.905339 16385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:57.948055 16385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:57.948464 16385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:57.948652 16385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:57.948936 16385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:57.949335 16385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:57.949399 16385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:57.949440 16385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:57.949486 16385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:57.956796 16385 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:33177
I20260811 11:19:57.956841 16525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:33177 every 8 connection(s)
I20260811 11:19:57.957254 16385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:19:57.957732 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 16385
I20260811 11:19:57.957855 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:19:57.958108 28772 external_mini_cluster.cc:1428] Setting key e12b3a70506c08ea0150141d8913639c
I20260811 11:19:57.963507 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.958215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:38625 (local address 127.28.25.62:41823)
0811 11:19:57.958286 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.958288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.958616 (+ 328us) server_negotiation.cc:407] Connection header received
0811 11:19:57.959395 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.959396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.959407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.959495 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:57.960289 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.960835 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.961951 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.962165 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.962251 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.963221 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:57.963223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:57.963223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:19:57.963228 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:57.963244 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:19:57.963249 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:57.963309 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.963385 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.963420 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 11:19:57.963745 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.958192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:41823 (local address 127.28.25.3:38625)
0811 11:19:57.958449 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:57.958502 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:57.959260 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:57.959599 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:57.959603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:57.959612 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:57.960190 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.960195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.960940 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.960941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.961767 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.961769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.962262 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.962263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.962417 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.963058 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:19:57.963058 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:57.963090 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:19:57.963114 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:57.963314 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:57.963319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:57.963323 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:19:57.963390 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 11:19:57.963498 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":108,"threads_started":1}
I20260811 11:19:57.964267 16526 heartbeater.cc:347] Connected to a master server at 127.28.25.62:41823
I20260811 11:19:57.964359 16526 heartbeater.cc:464] Registering TS with master...
I20260811 11:19:57.964577 16526 heartbeater.cc:511] Master 127.28.25.62:41823 requested a full tablet report, sending...
I20260811 11:19:57.965022 16036 ts_manager.cc:194] Registered new tserver with Master: 8c7925f186e94d459e0ff46c6b624ac9 (127.28.25.3:33177)
I20260811 11:19:57.965401 16036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.3:38625
I20260811 11:19:57.970188 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:19:57.975442 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.971793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.28.25.62:41823)
0811 11:19:57.971916 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.971918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.972045 (+ 127us) server_negotiation.cc:407] Connection header received
0811 11:19:57.972139 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.972140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.972156 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.972215 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:57.972426 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.972996 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.973925 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.974145 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.974239 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.975117 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:57.975118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:57.975119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:19:57.975125 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:57.975142 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:19:57.975152 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:57.975217 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.975273 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.975302 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 11:19:57.977174 16036 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:19:57.978806 16036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:19:57.986330 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.981753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.28.25.2:36223)
0811 11:19:57.982029 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.982035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.982443 (+ 408us) server_negotiation.cc:407] Connection header received
0811 11:19:57.982551 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.982554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.982571 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.982654 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:57.982759 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.983963 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.985135 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.985820 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.985931 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.986050 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.986177 (+ 127us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.986223 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":122,"threads_started":1}
I20260811 11:19:57.986354 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.981398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:33127 (local address 127.0.0.1:41604)
0811 11:19:57.981513 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:57.981557 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:57.981627 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:57.982535 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:57.982535 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:57.982538 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:57.982593 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.982595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.983643 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.983645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.985053 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.985056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.986036 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.986037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.986110 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.986114 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:19:57.986147 (+ 33us) client_negotiation.cc:240] Negotiation successful
0811 11:19:57.986246 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":106,"thread_start_us":66,"threads_started":1}
I20260811 11:19:57.986466 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.982021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.28.25.1:33127)
0811 11:19:57.982309 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.982313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.982324 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:19:57.982365 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.982367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.982381 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.982714 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:57.982785 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.983554 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.985162 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.985936 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.986208 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.986307 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.986335 (+ 28us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.986377 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":124,"threads_started":1}
I20260811 11:19:57.986722 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.981404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:33177 (local address 127.0.0.1:56418)
0811 11:19:57.981720 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:57.981732 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:57.981791 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:57.982179 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:57.982184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:57.982186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:57.982271 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.982274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.983809 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.983810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.985053 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.985056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.986035 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.986036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.986112 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.986115 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:19:57.986147 (+ 32us) client_negotiation.cc:240] Negotiation successful
0811 11:19:57.986633 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":8,"thread_start_us":59,"threads_started":1}
I20260811 11:19:57.987159 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.981745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.28.25.3:33177)
0811 11:19:57.982004 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:57.982008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:57.982018 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:19:57.982052 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:57.982054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:57.982066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:57.982133 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:57.982374 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.983719 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.985164 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.985925 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.986834 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.986947 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:57.987014 (+ 67us) server_negotiation.cc:299] Negotiation successful
0811 11:19:57.987065 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":120,"threads_started":1}
I20260811 11:19:57.986359 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:57.981619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:36223 (local address 127.0.0.1:40128)
0811 11:19:57.982241 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:57.982254 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:57.982317 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:57.982662 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:57.982663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:57.982665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:57.982700 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.982701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.984056 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.984057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.985039 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:57.985042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:57.985936 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:57.985938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:57.986025 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:57.986042 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 11:19:57.986157 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 11:19:57.986255 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":44,"thread_start_us":47,"threads_started":1}
I20260811 11:19:57.987407 16164 tablet_service.cc:1467] Processing CreateTablet for tablet 4f16a1dbc1d340938b375aa4efb842b8 (DEFAULT_TABLE table=test-table [id=02ca5a37ec9e4c64bb82ed2c90e2e35d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:19:57.987843 16164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f16a1dbc1d340938b375aa4efb842b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:57.987823 16451 tablet_service.cc:1467] Processing CreateTablet for tablet 4f16a1dbc1d340938b375aa4efb842b8 (DEFAULT_TABLE table=test-table [id=02ca5a37ec9e4c64bb82ed2c90e2e35d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:19:57.988175 16451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f16a1dbc1d340938b375aa4efb842b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:57.988083 16309 tablet_service.cc:1467] Processing CreateTablet for tablet 4f16a1dbc1d340938b375aa4efb842b8 (DEFAULT_TABLE table=test-table [id=02ca5a37ec9e4c64bb82ed2c90e2e35d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:19:57.988381 16309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f16a1dbc1d340938b375aa4efb842b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:57.990733 16542 tablet_bootstrap.cc:492] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794: Bootstrap starting.
I20260811 11:19:57.991498 16542 tablet_bootstrap.cc:654] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:57.991724 16546 tablet_bootstrap.cc:492] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: Bootstrap starting.
I20260811 11:19:57.992054 16542 log.cc:824] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:57.992475 16546 tablet_bootstrap.cc:654] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:57.992460 16544 tablet_bootstrap.cc:492] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: Bootstrap starting.
I20260811 11:19:57.992898 16542 tablet_bootstrap.cc:492] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794: No bootstrap required, opened a new log
I20260811 11:19:57.992892 16546 log.cc:824] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:57.992996 16542 ts_tablet_manager.cc:1397] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:19:57.993222 16544 tablet_bootstrap.cc:654] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:57.993597 16544 log.cc:824] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:57.993916 16546 tablet_bootstrap.cc:492] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: No bootstrap required, opened a new log
I20260811 11:19:57.994097 16546 ts_tablet_manager.cc:1397] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:19:57.994433 16544 tablet_bootstrap.cc:492] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: No bootstrap required, opened a new log
I20260811 11:19:57.994508 16544 ts_tablet_manager.cc:1397] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:19:57.995535 16542 raft_consensus.cc:348] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 } } peers { permanent_uuid: "c7114d04379143b1b61d32964d6ac794" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 33127 } } peers { permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 } }
I20260811 11:19:57.995700 16542 raft_consensus.cc:374] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:57.995746 16542 raft_consensus.cc:729] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7114d04379143b1b61d32964d6ac794, State: Initialized, Role: FOLLOWER
I20260811 11:19:57.995864 16542 consensus_queue.cc:260] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 } } peers { permanent_uuid: "c7114d04379143b1b61d32964d6ac794" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 33127 } } peers { permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 } }
I20260811 11:19:57.996176 16542 ts_tablet_manager.cc:1428] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:19:57.996196 16234 heartbeater.cc:503] Master 127.28.25.62:41823 was elected leader, sending a full tablet report...
I20260811 11:19:57.997502 16544 raft_consensus.cc:348] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 } } peers { permanent_uuid: "c7114d04379143b1b61d32964d6ac794" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 33127 } } peers { permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 } }
I20260811 11:19:57.997581 16546 raft_consensus.cc:348] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 } } peers { permanent_uuid: "c7114d04379143b1b61d32964d6ac794" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 33127 } } peers { permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 } }
I20260811 11:19:57.997692 16544 raft_consensus.cc:374] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:57.997730 16546 raft_consensus.cc:374] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:57.997732 16544 raft_consensus.cc:729] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516786f85cae43e9b577482b58fcbebb, State: Initialized, Role: FOLLOWER
I20260811 11:19:57.997776 16546 raft_consensus.cc:729] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7925f186e94d459e0ff46c6b624ac9, State: Initialized, Role: FOLLOWER
I20260811 11:19:57.997888 16546 consensus_queue.cc:260] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 } } peers { permanent_uuid: "c7114d04379143b1b61d32964d6ac794" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 33127 } } peers { permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 } }
I20260811 11:19:57.997879 16544 consensus_queue.cc:260] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 } } peers { permanent_uuid: "c7114d04379143b1b61d32964d6ac794" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 33127 } } peers { permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 } }
I20260811 11:19:57.998188 16546 ts_tablet_manager.cc:1428] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260811 11:19:57.998186 16544 ts_tablet_manager.cc:1428] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260811 11:19:57.998234 16382 heartbeater.cc:503] Master 127.28.25.62:41823 was elected leader, sending a full tablet report...
I20260811 11:19:57.998315 16526 heartbeater.cc:503] Master 127.28.25.62:41823 was elected leader, sending a full tablet report...
W20260811 11:19:58.040426 16383 tablet.cc:2369] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:19:58.092553 16550 raft_consensus.cc:482] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:19:58.092842 16550 raft_consensus.cc:504] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 } } peers { permanent_uuid: "c7114d04379143b1b61d32964d6ac794" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 33127 } } peers { permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 } }
I20260811 11:19:58.093257 16550 leader_election.cc:288] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c7925f186e94d459e0ff46c6b624ac9 (127.28.25.3:33177), 516786f85cae43e9b577482b58fcbebb (127.28.25.2:36223)
I20260811 11:19:58.096259 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:58.093470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:33177 (local address 127.28.25.1:51825)
0811 11:19:58.093539 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:58.093552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:58.093602 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:58.093751 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:58.093752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:58.093754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:58.093878 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:58.093885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:58.094552 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:58.094553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:58.095409 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:58.095412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:58.095990 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:58.095991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:58.096065 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:58.096068 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:19:58.096133 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:19:58.096181 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 11:19:58.096259 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:58.093525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:51825 (local address 127.28.25.3:33177)
0811 11:19:58.093570 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:58.093572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:58.093587 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:19:58.093665 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:58.093666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:58.093676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:58.093736 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:58.093968 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:58.094461 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:58.095505 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:58.095898 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:58.095987 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:58.095997 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:58.096153 (+ 156us) server_negotiation.cc:299] Negotiation successful
0811 11:19:58.096193 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:19:58.096410 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:58.093577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:34465 (local address 127.28.25.2:36223)
0811 11:19:58.093626 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:58.093628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:58.093796 (+ 168us) server_negotiation.cc:407] Connection header received
0811 11:19:58.093878 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:58.093879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:58.093892 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:58.093967 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:19:58.094122 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:58.094599 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:58.095655 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:58.096030 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:58.096116 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:58.096124 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:58.096305 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 11:19:58.096348 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:19:58.096414 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:58.093511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:36223 (local address 127.28.25.1:34465)
0811 11:19:58.093710 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:58.093719 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:58.093808 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:58.093972 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:58.093973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:58.093974 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:19:58.094050 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:58.094051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:58.094688 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:58.094689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:58.095554 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:58.095556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:58.096120 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:58.096121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:58.096208 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:58.096211 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:19:58.096294 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 11:19:58.096332 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":123,"threads_started":1}
I20260811 11:19:58.096642 16476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f16a1dbc1d340938b375aa4efb842b8" candidate_uuid: "c7114d04379143b1b61d32964d6ac794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c7925f186e94d459e0ff46c6b624ac9" is_pre_election: true
I20260811 11:19:58.096839 16476 raft_consensus.cc:2454] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7114d04379143b1b61d32964d6ac794 in term 0.
I20260811 11:19:58.096880 16331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f16a1dbc1d340938b375aa4efb842b8" candidate_uuid: "c7114d04379143b1b61d32964d6ac794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516786f85cae43e9b577482b58fcbebb" is_pre_election: true
I20260811 11:19:58.097100 16331 raft_consensus.cc:2454] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7114d04379143b1b61d32964d6ac794 in term 0.
I20260811 11:19:58.097129 16116 leader_election.cc:302] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c7925f186e94d459e0ff46c6b624ac9, c7114d04379143b1b61d32964d6ac794; no voters:
I20260811 11:19:58.097306 16550 raft_consensus.cc:2781] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:19:58.097363 16550 raft_consensus.cc:482] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:19:58.097399 16550 raft_consensus.cc:3037] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:19:58.098140 16550 raft_consensus.cc:504] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 } } peers { permanent_uuid: "c7114d04379143b1b61d32964d6ac794" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 33127 } } peers { permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 } }
I20260811 11:19:58.098311 16550 leader_election.cc:288] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [CANDIDATE]: Term 1 election: Requested vote from peers 8c7925f186e94d459e0ff46c6b624ac9 (127.28.25.3:33177), 516786f85cae43e9b577482b58fcbebb (127.28.25.2:36223)
I20260811 11:19:58.098482 16331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f16a1dbc1d340938b375aa4efb842b8" candidate_uuid: "c7114d04379143b1b61d32964d6ac794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516786f85cae43e9b577482b58fcbebb"
I20260811 11:19:58.098485 16476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f16a1dbc1d340938b375aa4efb842b8" candidate_uuid: "c7114d04379143b1b61d32964d6ac794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c7925f186e94d459e0ff46c6b624ac9"
I20260811 11:19:58.098567 16331 raft_consensus.cc:3037] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:19:58.098590 16476 raft_consensus.cc:3037] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:19:58.099473 16476 raft_consensus.cc:2454] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7114d04379143b1b61d32964d6ac794 in term 1.
I20260811 11:19:58.099488 16331 raft_consensus.cc:2454] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7114d04379143b1b61d32964d6ac794 in term 1.
I20260811 11:19:58.099658 16116 leader_election.cc:302] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c7925f186e94d459e0ff46c6b624ac9, c7114d04379143b1b61d32964d6ac794; no voters:
I20260811 11:19:58.099803 16550 raft_consensus.cc:2781] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:19:58.100013 16550 raft_consensus.cc:686] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 1 LEADER]: Becoming Leader. State: Replica: c7114d04379143b1b61d32964d6ac794, State: Running, Role: LEADER
I20260811 11:19:58.100119 16550 consensus_queue.cc:237] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 } } peers { permanent_uuid: "c7114d04379143b1b61d32964d6ac794" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 33127 } } peers { permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 } }
I20260811 11:19:58.101051 16036 catalog_manager.cc:5495] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7114d04379143b1b61d32964d6ac794 (127.28.25.1). New cstate: current_term: 1 leader_uuid: "c7114d04379143b1b61d32964d6ac794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7114d04379143b1b61d32964d6ac794" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 33127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260811 11:19:58.106792 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:58.104280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41606 (local address 127.28.25.1:33127)
0811 11:19:58.104347 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:58.104349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:58.104360 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:19:58.104428 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:58.104428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:58.104449 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:58.104535 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:19:58.104693 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:58.105174 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:58.105944 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:58.106153 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:58.106232 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:58.106410 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:19:58.106516 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:19:58.106625 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:58.106645 (+ 20us) server_negotiation.cc:299] Negotiation successful
0811 11:19:58.106667 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:19:58.108789 16331 raft_consensus.cc:1261] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb [term 1 FOLLOWER]: Refusing update from remote peer c7114d04379143b1b61d32964d6ac794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:19:58.108798 16476 raft_consensus.cc:1261] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9 [term 1 FOLLOWER]: Refusing update from remote peer c7114d04379143b1b61d32964d6ac794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:19:58.109238 16554 consensus_queue.cc:1035] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516786f85cae43e9b577482b58fcbebb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:19:58.109380 16550 consensus_queue.cc:1035] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7925f186e94d459e0ff46c6b624ac9" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:19:58.112059 16559 mvcc.cc:205] Tried to move back new op lower bound from 7317287723450568704 to 7317287723418705920. Current Snapshot: MvccSnapshot[applied={T|T < 7317287723450568704}]
I20260811 11:19:58.112668 16560 mvcc.cc:205] Tried to move back new op lower bound from 7317287723450568704 to 7317287723418705920. Current Snapshot: MvccSnapshot[applied={T|T < 7317287723450568704}]
I20260811 11:19:58.114383 16035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:19:58.114498 16035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:19:58.115293 16035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b59ed05832f84cdcb4dee7a07fddd32b: Sending DeleteTablet for 3 replicas of tablet 4f16a1dbc1d340938b375aa4efb842b8
I20260811 11:19:58.115657 16309 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f16a1dbc1d340938b375aa4efb842b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:19:58 UTC) from {username='slave'} at 127.0.0.1:40128
I20260811 11:19:58.115665 16164 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f16a1dbc1d340938b375aa4efb842b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:19:58 UTC) from {username='slave'} at 127.0.0.1:41604
I20260811 11:19:58.115701 16451 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f16a1dbc1d340938b375aa4efb842b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:19:58 UTC) from {username='slave'} at 127.0.0.1:56418
I20260811 11:19:58.115890 16569 tablet_replica.cc:330] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794: stopping tablet replica
I20260811 11:19:58.115995 16569 raft_consensus.cc:2229] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:19:58.116115 16569 raft_consensus.cc:2258] T 4f16a1dbc1d340938b375aa4efb842b8 P c7114d04379143b1b61d32964d6ac794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:19:58.116191 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 16099
I20260811 11:19:58.116762 16570 tablet_replica.cc:330] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: stopping tablet replica
I20260811 11:19:58.116868 16570 raft_consensus.cc:2229] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:19:58.116947 16570 raft_consensus.cc:2258] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:19:58.117486 16571 tablet_replica.cc:330] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: stopping tablet replica
I20260811 11:19:58.117594 16571 raft_consensus.cc:2229] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:19:58.117693 16571 raft_consensus.cc:2258] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:19:58.117764 16570 ts_tablet_manager.cc:1905] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:19:58.119671 16571 ts_tablet_manager.cc:1905] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:19:58.120081 16570 ts_tablet_manager.cc:1918] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:19:58.120198 16570 log.cc:1196] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/4f16a1dbc1d340938b375aa4efb842b8
I20260811 11:19:58.120458 16570 ts_tablet_manager.cc:1939] T 4f16a1dbc1d340938b375aa4efb842b8 P 516786f85cae43e9b577482b58fcbebb: Deleting consensus metadata
I20260811 11:19:58.120932 16018 catalog_manager.cc:4841] TS 516786f85cae43e9b577482b58fcbebb (127.28.25.2:36223): tablet 4f16a1dbc1d340938b375aa4efb842b8 (table test-table [id=02ca5a37ec9e4c64bb82ed2c90e2e35d]) successfully deleted
I20260811 11:19:58.121838 16571 ts_tablet_manager.cc:1918] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:19:58.121930 16571 log.cc:1196] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/4f16a1dbc1d340938b375aa4efb842b8
I20260811 11:19:58.122184 16571 ts_tablet_manager.cc:1939] T 4f16a1dbc1d340938b375aa4efb842b8 P 8c7925f186e94d459e0ff46c6b624ac9: Deleting consensus metadata
I20260811 11:19:58.123001 16020 catalog_manager.cc:4841] TS 8c7925f186e94d459e0ff46c6b624ac9 (127.28.25.3:33177): tablet 4f16a1dbc1d340938b375aa4efb842b8 (table test-table [id=02ca5a37ec9e4c64bb82ed2c90e2e35d]) successfully deleted
W20260811 11:19:58.124754 16018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:33127 (error 108)
I20260811 11:19:58.125324 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:58.125131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:33127 (local address 127.0.0.1:41610)
0811 11:19:58.125194 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:58.125247 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:33127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260811 11:19:58.125381 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 16239
W20260811 11:19:58.125476 16018 catalog_manager.cc:4584] TS c7114d04379143b1b61d32964d6ac794 (127.28.25.1:33127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f16a1dbc1d340938b375aa4efb842b8: Network error: Client connection negotiation failed: client connection to 127.28.25.1:33127: connect: Connection refused (error 111)
I20260811 11:19:58.135564 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 16385
I20260811 11:19:58.145165 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 16003
2026-08-11T11:19:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (846 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:19:58 dist-test-slave-2w3w krb5kdc[16576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:19:58 dist-test-slave-2w3w krb5kdc[16576](info): set up 2 sockets
Aug 11 11:19:58 dist-test-slave-2w3w krb5kdc[16576](info): commencing operation
krb5kdc: starting...
W20260811 11:19:59.315534 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:19:59 dist-test-slave-2w3w krb5kdc[16576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:19:59.451263 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:37527
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:37527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:19:59.538340 16589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:59.538599 16589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:59.538642 16589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:59.540001 16589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:19:59.540086 16589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:59.540112 16589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:59.540144 16589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:19:59.540169 16589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:19:59.541838 16589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:37527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:37527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:59.542157 16589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:59.542413 16589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:59.547981 16595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:59.548018 16598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:59.547981 16596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:59.548302 16589 server_base.cc:1034] running on GCE node
I20260811 11:19:59.548556 16589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:59.548825 16589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:59.549969 16589 hybrid_clock.cc:648] HybridClock initialized: now 1786447199549948 us; error 38 us; skew 500 ppm
Aug 11 11:19:59 dist-test-slave-2w3w krb5kdc[16576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447199, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:59.552676 16589 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:19:59.553222 16589 webserver.cc:460] Webserver started at http://127.28.25.62:36739/ using document root <none> and password file <none>
I20260811 11:19:59.553483 16589 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:59.553555 16589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:59.553711 16589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:59.554832 16589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "458ae28622ca4d67bdef71361150d327"
format_stamp: "Formatted at 2026-08-11 11:19:59 on dist-test-slave-2w3w"
server_key: "9760a57a11a002b27d9160908c171dfc"
server_key_iv: "7a572d6417484e8cb56ae286a9d1832b"
server_key_version: "encryptionkey@0"
I20260811 11:19:59.555265 16589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "458ae28622ca4d67bdef71361150d327"
format_stamp: "Formatted at 2026-08-11 11:19:59 on dist-test-slave-2w3w"
server_key: "9760a57a11a002b27d9160908c171dfc"
server_key_iv: "7a572d6417484e8cb56ae286a9d1832b"
server_key_version: "encryptionkey@0"
I20260811 11:19:59.556605 16589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:19:59.557421 16606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:59.557591 16589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260811 11:19:59.557700 16589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "458ae28622ca4d67bdef71361150d327"
format_stamp: "Formatted at 2026-08-11 11:19:59 on dist-test-slave-2w3w"
server_key: "9760a57a11a002b27d9160908c171dfc"
server_key_iv: "7a572d6417484e8cb56ae286a9d1832b"
server_key_version: "encryptionkey@0"
I20260811 11:19:59.557825 16589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:59.581260 16589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:59.583616 16589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:59.583823 16589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:59.588630 16589 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:37527
I20260811 11:19:59.588667 16672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:37527 every 8 connection(s)
I20260811 11:19:59.589079 16589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:19:59.589999 16673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:19:59.592465 16673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327: Bootstrap starting.
I20260811 11:19:59.593061 16673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327: Neither blocks nor log segments found. Creating new log.
I20260811 11:19:59.593310 16673 log.cc:824] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327: Log is configured to *not* fsync() on all Append() calls
I20260811 11:19:59.594091 16673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327: No bootstrap required, opened a new log
I20260811 11:19:59.596203 16673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ae28622ca4d67bdef71361150d327" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37527 } }
I20260811 11:19:59.596379 16673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:19:59.596403 16673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458ae28622ca4d67bdef71361150d327, State: Initialized, Role: FOLLOWER
I20260811 11:19:59.596560 16673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ae28622ca4d67bdef71361150d327" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37527 } }
I20260811 11:19:59.596647 16673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:19:59.596670 16673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:19:59.596701 16673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:19:59.597285 16673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ae28622ca4d67bdef71361150d327" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37527 } }
I20260811 11:19:59.597380 16673 leader_election.cc:302] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 458ae28622ca4d67bdef71361150d327; no voters:
I20260811 11:19:59.597599 16673 leader_election.cc:288] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:19:59.597661 16676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:19:59.597719 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 16589
I20260811 11:19:59.597848 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:19:59.597918 16673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:19:59.597925 16676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [term 1 LEADER]: Becoming Leader. State: Replica: 458ae28622ca4d67bdef71361150d327, State: Running, Role: LEADER
I20260811 11:19:59.598089 28772 external_mini_cluster.cc:1428] Setting key bd4a8f503b8a289857bb4abaa63d37d6
I20260811 11:19:59.598083 16676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ae28622ca4d67bdef71361150d327" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37527 } }
I20260811 11:19:59.598445 16678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 458ae28622ca4d67bdef71361150d327. Latest consensus state: current_term: 1 leader_uuid: "458ae28622ca4d67bdef71361150d327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ae28622ca4d67bdef71361150d327" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37527 } } }
I20260811 11:19:59.598541 16678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [sys.catalog]: This master's current role is: LEADER
I20260811 11:19:59.598692 16677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "458ae28622ca4d67bdef71361150d327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ae28622ca4d67bdef71361150d327" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 37527 } } }
I20260811 11:19:59.598766 16681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:19:59.598850 16677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327 [sys.catalog]: This master's current role is: LEADER
I20260811 11:19:59.599712 16681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:19:59.601505 16681 catalog_manager.cc:1347] Generated new cluster ID: c80995ba32b44a73a58309faa4c625ac
I20260811 11:19:59.601562 16681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 11:19:59 dist-test-slave-2w3w krb5kdc[16576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:19:59.611624 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:59.598636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.28.25.62:37527)
0811 11:19:59.599137 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:59.599149 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:59.599195 (+ 46us) server_negotiation.cc:407] Connection header received
0811 11:19:59.599885 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:59.599892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:59.599963 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:59.600972 (+ 1009us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:59.601089 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:59.602500 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:59.603622 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:59.604319 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:59.604442 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:59.607536 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:59.607542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:59.607542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:59.607556 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:59.609676 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:59.610148 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:59.610151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:59.610151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:59.610284 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:59.610521 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:59.610521 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:59.610522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:59.610856 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:59.611051 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:59.611194 (+ 143us) server_negotiation.cc:299] Negotiation successful
0811 11:19:59.611347 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":67,"threads_started":1}
I20260811 11:19:59.612417 16681 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:19:59.613142 16681 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:19:59.623903 16681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327: Generated new TSK 0
I20260811 11:19:59.624159 16681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:19:59.662312 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37527
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:19:59.748730 16702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:59.748956 16702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:59.748986 16702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:59.750145 16702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:59.750217 16702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:59.750254 16702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:19:59.751746 16702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:59.751950 16702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:59.752139 16702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:59.757579 16707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:59.757627 16708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:59.757642 16711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:59.757723 16702 server_base.cc:1034] running on GCE node
I20260811 11:19:59.758201 16702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:59.758457 16702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:59.759627 16702 hybrid_clock.cc:648] HybridClock initialized: now 1786447199759612 us; error 46 us; skew 500 ppm
Aug 11 11:19:59 dist-test-slave-2w3w krb5kdc[16576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447199, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:59.762506 16702 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:19:59.763077 16702 webserver.cc:460] Webserver started at http://127.28.25.1:41247/ using document root <none> and password file <none>
I20260811 11:19:59.763278 16702 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:59.763320 16702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:59.763413 16702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:59.764442 16702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "11c79a74a7b54854a4d0598eeed28be7"
format_stamp: "Formatted at 2026-08-11 11:19:59 on dist-test-slave-2w3w"
server_key: "fe3e6f08d00d15bc90b0c8137cb57e8e"
server_key_iv: "4031188ad183d2a34f6983160158eb30"
server_key_version: "encryptionkey@0"
I20260811 11:19:59.764753 16702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "11c79a74a7b54854a4d0598eeed28be7"
format_stamp: "Formatted at 2026-08-11 11:19:59 on dist-test-slave-2w3w"
server_key: "fe3e6f08d00d15bc90b0c8137cb57e8e"
server_key_iv: "4031188ad183d2a34f6983160158eb30"
server_key_version: "encryptionkey@0"
I20260811 11:19:59.766176 16702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:19:59.767050 16719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:59.767263 16702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:19:59.767395 16702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "11c79a74a7b54854a4d0598eeed28be7"
format_stamp: "Formatted at 2026-08-11 11:19:59 on dist-test-slave-2w3w"
server_key: "fe3e6f08d00d15bc90b0c8137cb57e8e"
server_key_iv: "4031188ad183d2a34f6983160158eb30"
server_key_version: "encryptionkey@0"
I20260811 11:19:59.767510 16702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:59.780375 16702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:59.782558 16702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:59.782748 16702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:59.783149 16702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:59.783525 16702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:59.783588 16702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:59.783635 16702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:59.783666 16702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:59.790522 16702 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:36655
I20260811 11:19:59.790580 16842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:36655 every 8 connection(s)
I20260811 11:19:59.790987 16702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
Aug 11 11:19:59 dist-test-slave-2w3w krb5kdc[16576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447199, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:19:59.798790 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 16702
I20260811 11:19:59.798985 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:19:59.799324 28772 external_mini_cluster.cc:1428] Setting key d4144522fa273f96ba9ae239569f54a4
I20260811 11:19:59.801875 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:59.791811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:57851 (local address 127.28.25.62:37527)
0811 11:19:59.791947 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:59.791949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:59.792167 (+ 218us) server_negotiation.cc:407] Connection header received
0811 11:19:59.792652 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:59.792653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:59.792670 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:59.792739 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:59.793773 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:59.794369 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:59.795435 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:59.795652 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:59.795715 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:59.798160 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:59.798162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:59.798162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:59.798170 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:59.800067 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:59.800560 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:59.800562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:59.800562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:59.800633 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:59.800902 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:59.800903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:59.800903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:59.801073 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:59.801153 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:19:59.801594 (+ 441us) server_negotiation.cc:299] Negotiation successful
0811 11:19:59.801704 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:19:59.801914 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:59.791796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37527 (local address 127.28.25.1:57851)
0811 11:19:59.791992 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:59.792039 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:59.792558 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:59.792808 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:59.792813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:59.793465 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:59.793672 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:59.793677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:59.794465 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:59.794466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:59.795354 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:59.795356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:59.795737 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:59.795738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:59.795866 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:59.796462 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:19:59.796470 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:59.798058 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:59.800176 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:59.800186 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:59.800189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:59.800477 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:59.800705 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:59.800706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:59.800706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:59.800831 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:59.801205 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:59.801222 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:19:59.801433 (+ 211us) client_negotiation.cc:769] Sending connection context
0811 11:19:59.801539 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 11:19:59.801693 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
I20260811 11:19:59.802950 16843 heartbeater.cc:347] Connected to a master server at 127.28.25.62:37527
I20260811 11:19:59.803154 16843 heartbeater.cc:464] Registering TS with master...
I20260811 11:19:59.803396 16843 heartbeater.cc:511] Master 127.28.25.62:37527 requested a full tablet report, sending...
I20260811 11:19:59.804037 16633 ts_manager.cc:194] Registered new tserver with Master: 11c79a74a7b54854a4d0598eeed28be7 (127.28.25.1:36655)
I20260811 11:19:59.804982 16633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:57851
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:19:59.830806 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37527
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:19:59.926535 16848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:19:59.926783 16848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:19:59.926813 16848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:19:59.928207 16848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:19:59.928264 16848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:19:59.928303 16848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:19:59.930083 16848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:19:59.930395 16848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:19:59.930680 16848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:19:59.936409 16858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:59.936489 16854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:19:59.936403 16855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:19:59.936622 16848 server_base.cc:1034] running on GCE node
I20260811 11:19:59.936830 16848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:19:59.937120 16848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:19:59.938310 16848 hybrid_clock.cc:648] HybridClock initialized: now 1786447199938289 us; error 43 us; skew 500 ppm
Aug 11 11:19:59 dist-test-slave-2w3w krb5kdc[16576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447199, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:19:59.950256 16848 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:19:59.950923 16848 webserver.cc:460] Webserver started at http://127.28.25.2:37247/ using document root <none> and password file <none>
I20260811 11:19:59.951261 16848 fs_manager.cc:359] Metadata directory not provided
I20260811 11:19:59.951359 16848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:19:59.951535 16848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:19:59.952783 16848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "e150f1d004b543b192ff403e505878a6"
format_stamp: "Formatted at 2026-08-11 11:19:59 on dist-test-slave-2w3w"
server_key: "e639349afb884e7f4aa4d69bbcf322ff"
server_key_iv: "a55722d0003b5ac16b3e7efd17252f16"
server_key_version: "encryptionkey@0"
I20260811 11:19:59.953223 16848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "e150f1d004b543b192ff403e505878a6"
format_stamp: "Formatted at 2026-08-11 11:19:59 on dist-test-slave-2w3w"
server_key: "e639349afb884e7f4aa4d69bbcf322ff"
server_key_iv: "a55722d0003b5ac16b3e7efd17252f16"
server_key_version: "encryptionkey@0"
I20260811 11:19:59.954948 16848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:19:59.955921 16865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:59.956164 16848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:19:59.956300 16848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "e150f1d004b543b192ff403e505878a6"
format_stamp: "Formatted at 2026-08-11 11:19:59 on dist-test-slave-2w3w"
server_key: "e639349afb884e7f4aa4d69bbcf322ff"
server_key_iv: "a55722d0003b5ac16b3e7efd17252f16"
server_key_version: "encryptionkey@0"
I20260811 11:19:59.956395 16848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:19:59.977293 16848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:19:59.980314 16848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:19:59.980463 16848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:19:59.980674 16848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:19:59.981055 16848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:19:59.981096 16848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:59.981119 16848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:19:59.981137 16848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:19:59.987778 16848 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:41063
I20260811 11:19:59.987933 16989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:41063 every 8 connection(s)
I20260811 11:19:59.988240 16848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
Aug 11 11:19:59 dist-test-slave-2w3w krb5kdc[16576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447199, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:19:59.998328 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 16848
I20260811 11:19:59.998493 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:19:59.998733 28772 external_mini_cluster.cc:1428] Setting key cc131eb0d1a26455608efcb196d908d5
I20260811 11:20:00.001158 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:59.989121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:51723 (local address 127.28.25.62:37527)
0811 11:19:59.989208 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0811 11:19:59.989210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:19:59.989447 (+ 237us) server_negotiation.cc:407] Connection header received
0811 11:19:59.990089 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:19:59.990091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:19:59.990103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:19:59.990175 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:19:59.991467 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:59.992084 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:59.993273 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:19:59.993489 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:19:59.993560 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:59.996456 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:19:59.996459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:19:59.996459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:19:59.996468 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:19:59.998307 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:59.999186 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:59.999188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:59.999189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:59.999311 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:19:59.999687 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:19:59.999688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:19:59.999689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:19:59.999852 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:19:59.999934 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.000893 (+ 959us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.001008 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:20:00.001309 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:19:59.989054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37527 (local address 127.28.25.2:51723)
0811 11:19:59.989286 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 11:19:59.989328 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 11:19:59.989893 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:19:59.990308 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:19:59.990312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:19:59.991103 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:19:59.991331 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:59.991337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:59.992210 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:59.992211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:59.993153 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:19:59.993156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:19:59.993606 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:19:59.993607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:19:59.993748 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:19:59.994372 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:19:59.994381 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:19:59.996317 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:19:59.998443 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:59.998458 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:59.998461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:59.999066 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:59.999398 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:19:59.999398 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:19:59.999399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:19:59.999622 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:19:59.999938 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:19:59.999945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:00.000183 (+ 238us) client_negotiation.cc:769] Sending connection context
0811 11:20:00.000797 (+ 614us) client_negotiation.cc:240] Negotiation successful
0811 11:20:00.001056 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":116,"threads_started":1}
I20260811 11:20:00.001874 16990 heartbeater.cc:347] Connected to a master server at 127.28.25.62:37527
I20260811 11:20:00.002019 16990 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:00.002247 16990 heartbeater.cc:511] Master 127.28.25.62:37527 requested a full tablet report, sending...
I20260811 11:20:00.002760 16633 ts_manager.cc:194] Registered new tserver with Master: e150f1d004b543b192ff403e505878a6 (127.28.25.2:41063)
I20260811 11:20:00.003363 16633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:51723
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:20:00.031164 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37527
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:00.118170 16996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:00.118450 16996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:00.118494 16996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:00.119856 16996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:00.119946 16996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:00.119997 16996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:20:00.121665 16996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:37527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:00.121981 16996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:00.122251 16996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:00.127907 17005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:00.127895 17001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:00.127902 17003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:00.128170 16996 server_base.cc:1034] running on GCE node
I20260811 11:20:00.128415 16996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:00.128706 16996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:00.129909 16996 hybrid_clock.cc:648] HybridClock initialized: now 1786447200129893 us; error 47 us; skew 500 ppm
Aug 11 11:20:00 dist-test-slave-2w3w krb5kdc[16576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:00.132645 16996 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:20:00.133138 16996 webserver.cc:460] Webserver started at http://127.28.25.3:39445/ using document root <none> and password file <none>
I20260811 11:20:00.133342 16996 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:00.133385 16996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:00.133482 16996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:00.134545 16996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "e11f4d622bed433998589cfec6d25c15"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "4ab814b0e6e1e333353452893e56674c"
server_key_iv: "2a4578ad76a9c985d32d72927dc8ce90"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.134888 16996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "e11f4d622bed433998589cfec6d25c15"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "4ab814b0e6e1e333353452893e56674c"
server_key_iv: "2a4578ad76a9c985d32d72927dc8ce90"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.136416 16996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:00.137303 17012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:00.137534 16996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:00.137635 16996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "e11f4d622bed433998589cfec6d25c15"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "4ab814b0e6e1e333353452893e56674c"
server_key_iv: "2a4578ad76a9c985d32d72927dc8ce90"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.137702 16996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:00.152093 16996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:00.154654 16996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:00.154803 16996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:00.155090 16996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:20:00.155404 16996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:20:00.155435 16996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:00.155459 16996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:20:00.155472 16996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:00.162496 16996 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:40299
I20260811 11:20:00.162590 17135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:40299 every 8 connection(s)
I20260811 11:20:00.162940 16996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:20:00.167861 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 16996
I20260811 11:20:00.168009 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:20:00.168251 28772 external_mini_cluster.cc:1428] Setting key 60923e9acccbc9191f1e78a3147c4d66
Aug 11 11:20:00 dist-test-slave-2w3w krb5kdc[16576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:00.173998 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.163882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:41717 (local address 127.28.25.62:37527)
0811 11:20:00.164054 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:00.164056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:00.164256 (+ 200us) server_negotiation.cc:407] Connection header received
0811 11:20:00.164843 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:00.164845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:00.164858 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:00.164914 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:00.165991 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.166510 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.167676 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.167884 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.167970 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.170773 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:00.170775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:00.170775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:00.170785 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:00.172375 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:00.172893 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:00.172895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:00.172895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:00.172966 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:00.173256 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:00.173257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:00.173257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:00.173388 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:00.173483 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.173794 (+ 311us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.173894 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260811 11:20:00.174249 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.163814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:37527 (local address 127.28.25.3:41717)
0811 11:20:00.164096 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:00.164154 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:00.164721 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:00.165034 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:00.165038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:00.165668 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:00.165876 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.165881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.166621 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.166623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.167552 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.167555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.167985 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.167986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.168170 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.168825 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:00.168836 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:00.170642 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:00.172521 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:00.172533 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:00.172536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:00.172805 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:00.173045 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:00.173046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:00.173046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:00.173176 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:00.173476 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:00.173483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:00.173650 (+ 167us) client_negotiation.cc:769] Sending connection context
0811 11:20:00.173770 (+ 120us) client_negotiation.cc:240] Negotiation successful
0811 11:20:00.173972 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
I20260811 11:20:00.174799 17136 heartbeater.cc:347] Connected to a master server at 127.28.25.62:37527
I20260811 11:20:00.174950 17136 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:00.175196 17136 heartbeater.cc:511] Master 127.28.25.62:37527 requested a full tablet report, sending...
I20260811 11:20:00.175740 16633 ts_manager.cc:194] Registered new tserver with Master: e11f4d622bed433998589cfec6d25c15 (127.28.25.3:40299)
I20260811 11:20:00.176182 16633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:41717
I20260811 11:20:00.181039 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:20:00.190399 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.182707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.28.25.62:37527)
0811 11:20:00.182796 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:00.182799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:00.182938 (+ 139us) server_negotiation.cc:407] Connection header received
0811 11:20:00.183043 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:00.183045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:00.183056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:00.183140 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:00.183801 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.184361 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.185217 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.185439 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.185523 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.187334 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:00.187336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:00.187336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:00.187346 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:00.188938 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:00.189444 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:00.189445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:00.189445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:00.189516 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:00.189752 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:00.189753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:00.189754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:00.189923 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:00.190045 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.190205 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.190286 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:20:00.192394 16633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:20:00.194105 16633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:20:00.201543 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.196896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.28.25.2:41063)
0811 11:20:00.197447 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:00.197452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:00.197463 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:20:00.197502 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:00.197504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:00.197518 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:00.197760 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:00.197854 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.198645 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.200332 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.201152 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.201238 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.201305 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.201436 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.201470 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":91,"threads_started":1}
I20260811 11:20:00.201555 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.196650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:41063 (local address 127.0.0.1:39790)
0811 11:20:00.197017 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:00.197065 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:00.197135 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:00.197622 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:00.197624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:00.197625 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:00.197678 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.197680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.198768 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.198769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.200226 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.200229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.201230 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.201231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.201325 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.201341 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 11:20:00.201411 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 11:20:00.201468 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":26,"thread_start_us":66,"threads_started":1}
I20260811 11:20:00.202227 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.196871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36655 (local address 127.0.0.1:36980)
0811 11:20:00.197258 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:00.197266 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:00.197316 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:00.197667 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:00.197668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:00.197670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:00.197743 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.197745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.199200 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.199201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.200328 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.200330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.201968 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.201970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.202033 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.202036 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:20:00.202093 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 11:20:00.202137 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":69,"threads_started":1}
I20260811 11:20:00.202431 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.196771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.28.25.3:40299)
0811 11:20:00.197062 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:00.197068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:00.197235 (+ 167us) server_negotiation.cc:407] Connection header received
0811 11:20:00.197298 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:00.197301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:00.197324 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:00.197404 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:00.197763 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.199081 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.201187 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.201943 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.202011 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.202069 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.202284 (+ 215us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.202338 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":148,"threads_started":1}
I20260811 11:20:00.202610 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.196650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:40299 (local address 127.0.0.1:34074)
0811 11:20:00.197069 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:00.197078 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:00.197135 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:00.197479 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:00.197482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:00.197485 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:00.197583 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.197587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.200412 (+ 2825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.200412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.201066 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.201068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.202100 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.202101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.202154 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.202157 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:20:00.202201 (+ 44us) client_negotiation.cc:240] Negotiation successful
0811 11:20:00.202231 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260811 11:20:00.203224 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.197013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.28.25.1:36655)
0811 11:20:00.197173 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:00.197176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:00.197326 (+ 150us) server_negotiation.cc:407] Connection header received
0811 11:20:00.197404 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:00.197406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:00.197423 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:00.197832 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:00.197946 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.199079 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.200405 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.201152 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.201243 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.201306 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.202135 (+ 829us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.202249 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260811 11:20:00.203449 17063 tablet_service.cc:1467] Processing CreateTablet for tablet ea1fcd924f4548c795f4e38f71ea820f (DEFAULT_TABLE table=test-table [id=9553ba0ddd48427ea535329c4199170b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:00.203622 16918 tablet_service.cc:1467] Processing CreateTablet for tablet ea1fcd924f4548c795f4e38f71ea820f (DEFAULT_TABLE table=test-table [id=9553ba0ddd48427ea535329c4199170b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:00.203843 17063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1fcd924f4548c795f4e38f71ea820f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:00.203998 16918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1fcd924f4548c795f4e38f71ea820f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:00.204041 16771 tablet_service.cc:1467] Processing CreateTablet for tablet ea1fcd924f4548c795f4e38f71ea820f (DEFAULT_TABLE table=test-table [id=9553ba0ddd48427ea535329c4199170b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:00.204339 16771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1fcd924f4548c795f4e38f71ea820f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:00.206529 17156 tablet_bootstrap.cc:492] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: Bootstrap starting.
I20260811 11:20:00.207243 17156 tablet_bootstrap.cc:654] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:00.207454 17157 tablet_bootstrap.cc:492] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6: Bootstrap starting.
I20260811 11:20:00.207891 17156 log.cc:824] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:00.208135 17162 tablet_bootstrap.cc:492] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7: Bootstrap starting.
I20260811 11:20:00.208178 17157 tablet_bootstrap.cc:654] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:00.208766 17162 tablet_bootstrap.cc:654] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:00.208789 17156 tablet_bootstrap.cc:492] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: No bootstrap required, opened a new log
I20260811 11:20:00.208848 17157 log.cc:824] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:00.208959 17156 ts_tablet_manager.cc:1397] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:20:00.209335 17162 log.cc:824] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:00.210196 17162 tablet_bootstrap.cc:492] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7: No bootstrap required, opened a new log
I20260811 11:20:00.210335 17162 ts_tablet_manager.cc:1397] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:20:00.210708 17156 raft_consensus.cc:348] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 } } peers { permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 } } peers { permanent_uuid: "e150f1d004b543b192ff403e505878a6" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 41063 } }
I20260811 11:20:00.210857 17156 raft_consensus.cc:374] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:00.210881 17156 raft_consensus.cc:729] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11f4d622bed433998589cfec6d25c15, State: Initialized, Role: FOLLOWER
I20260811 11:20:00.211035 17156 consensus_queue.cc:260] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 } } peers { permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 } } peers { permanent_uuid: "e150f1d004b543b192ff403e505878a6" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 41063 } }
I20260811 11:20:00.211364 17156 ts_tablet_manager.cc:1428] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260811 11:20:00.211403 17136 heartbeater.cc:503] Master 127.28.25.62:37527 was elected leader, sending a full tablet report...
I20260811 11:20:00.212069 17157 tablet_bootstrap.cc:492] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6: No bootstrap required, opened a new log
I20260811 11:20:00.212157 17157 ts_tablet_manager.cc:1397] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260811 11:20:00.212850 17162 raft_consensus.cc:348] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 } } peers { permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 } } peers { permanent_uuid: "e150f1d004b543b192ff403e505878a6" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 41063 } }
I20260811 11:20:00.213057 17162 raft_consensus.cc:374] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:00.213088 17162 raft_consensus.cc:729] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c79a74a7b54854a4d0598eeed28be7, State: Initialized, Role: FOLLOWER
I20260811 11:20:00.213260 17162 consensus_queue.cc:260] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 } } peers { permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 } } peers { permanent_uuid: "e150f1d004b543b192ff403e505878a6" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 41063 } }
I20260811 11:20:00.213589 17162 ts_tablet_manager.cc:1428] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260811 11:20:00.213660 16843 heartbeater.cc:503] Master 127.28.25.62:37527 was elected leader, sending a full tablet report...
I20260811 11:20:00.214313 17157 raft_consensus.cc:348] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 } } peers { permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 } } peers { permanent_uuid: "e150f1d004b543b192ff403e505878a6" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 41063 } }
I20260811 11:20:00.214486 17157 raft_consensus.cc:374] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:00.214532 17157 raft_consensus.cc:729] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e150f1d004b543b192ff403e505878a6, State: Initialized, Role: FOLLOWER
I20260811 11:20:00.214654 17157 consensus_queue.cc:260] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 } } peers { permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 } } peers { permanent_uuid: "e150f1d004b543b192ff403e505878a6" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 41063 } }
I20260811 11:20:00.215025 17157 ts_tablet_manager.cc:1428] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:20:00.215075 16990 heartbeater.cc:503] Master 127.28.25.62:37527 was elected leader, sending a full tablet report...
W20260811 11:20:00.238916 16991 tablet.cc:2369] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:20:00.291874 16844 tablet.cc:2369] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:20:00.413713 17137 tablet.cc:2369] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:20:00.453477 17168 raft_consensus.cc:482] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:20:00.453616 17168 raft_consensus.cc:504] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 } } peers { permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 } } peers { permanent_uuid: "e150f1d004b543b192ff403e505878a6" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 41063 } }
I20260811 11:20:00.453956 17168 leader_election.cc:288] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e11f4d622bed433998589cfec6d25c15 (127.28.25.3:40299), 11c79a74a7b54854a4d0598eeed28be7 (127.28.25.1:36655)
I20260811 11:20:00.457486 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.454361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:36089 (local address 127.28.25.3:40299)
0811 11:20:00.454444 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:00.454447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:00.454458 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:20:00.454493 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:00.454494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:00.454507 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:00.454612 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:00.454843 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.455464 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.456615 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.457053 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.457113 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.457123 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.457358 (+ 235us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.457399 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 11:20:00.457490 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.454461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:42709 (local address 127.28.25.1:36655)
0811 11:20:00.454541 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:00.454544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:00.454718 (+ 174us) server_negotiation.cc:407] Connection header received
0811 11:20:00.454814 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:00.454815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:00.454834 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:00.454898 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:00.455108 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.455707 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.456700 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.457118 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.457196 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.457206 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.457367 (+ 161us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.457407 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:20:00.457532 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.454201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:40299 (local address 127.28.25.2:36089)
0811 11:20:00.454294 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:00.454309 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:00.454369 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:00.454620 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:00.454621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:00.454623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:00.454749 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.454751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.455554 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.455555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.456519 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.456522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.457145 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.457146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.457243 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.457247 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:20:00.457350 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 11:20:00.457403 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":5}
I20260811 11:20:00.457542 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.454372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36655 (local address 127.28.25.2:42709)
0811 11:20:00.454616 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:00.454628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:00.454694 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:00.454916 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:00.454917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:00.454918 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:00.455019 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.455020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.455838 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.455840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.456604 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.456607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.457209 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.457210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.457276 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.457279 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:20:00.457354 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 11:20:00.457399 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":168,"threads_started":1}
I20260811 11:20:00.457978 17084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1fcd924f4548c795f4e38f71ea820f" candidate_uuid: "e150f1d004b543b192ff403e505878a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e11f4d622bed433998589cfec6d25c15" is_pre_election: true
I20260811 11:20:00.458002 16793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1fcd924f4548c795f4e38f71ea820f" candidate_uuid: "e150f1d004b543b192ff403e505878a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c79a74a7b54854a4d0598eeed28be7" is_pre_election: true
I20260811 11:20:00.458117 17084 raft_consensus.cc:2454] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e150f1d004b543b192ff403e505878a6 in term 0.
I20260811 11:20:00.458174 16793 raft_consensus.cc:2454] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e150f1d004b543b192ff403e505878a6 in term 0.
I20260811 11:20:00.458443 16868 leader_election.cc:302] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c79a74a7b54854a4d0598eeed28be7, e150f1d004b543b192ff403e505878a6; no voters:
I20260811 11:20:00.458581 17168 raft_consensus.cc:2781] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:20:00.458643 17168 raft_consensus.cc:482] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:20:00.458671 17168 raft_consensus.cc:3037] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:00.459539 17168 raft_consensus.cc:504] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 } } peers { permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 } } peers { permanent_uuid: "e150f1d004b543b192ff403e505878a6" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 41063 } }
I20260811 11:20:00.459679 17168 leader_election.cc:288] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [CANDIDATE]: Term 1 election: Requested vote from peers e11f4d622bed433998589cfec6d25c15 (127.28.25.3:40299), 11c79a74a7b54854a4d0598eeed28be7 (127.28.25.1:36655)
I20260811 11:20:00.459889 17084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1fcd924f4548c795f4e38f71ea820f" candidate_uuid: "e150f1d004b543b192ff403e505878a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e11f4d622bed433998589cfec6d25c15"
I20260811 11:20:00.459908 16793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1fcd924f4548c795f4e38f71ea820f" candidate_uuid: "e150f1d004b543b192ff403e505878a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c79a74a7b54854a4d0598eeed28be7"
I20260811 11:20:00.459981 17084 raft_consensus.cc:3037] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:00.459981 16793 raft_consensus.cc:3037] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:00.461004 16793 raft_consensus.cc:2454] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e150f1d004b543b192ff403e505878a6 in term 1.
I20260811 11:20:00.461004 17084 raft_consensus.cc:2454] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e150f1d004b543b192ff403e505878a6 in term 1.
I20260811 11:20:00.461201 16868 leader_election.cc:302] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c79a74a7b54854a4d0598eeed28be7, e150f1d004b543b192ff403e505878a6; no voters:
I20260811 11:20:00.461297 17168 raft_consensus.cc:2781] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:00.461417 17168 raft_consensus.cc:686] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 1 LEADER]: Becoming Leader. State: Replica: e150f1d004b543b192ff403e505878a6, State: Running, Role: LEADER
I20260811 11:20:00.461527 17168 consensus_queue.cc:237] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 } } peers { permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 } } peers { permanent_uuid: "e150f1d004b543b192ff403e505878a6" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 41063 } }
I20260811 11:20:00.462456 16633 catalog_manager.cc:5495] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e150f1d004b543b192ff403e505878a6 (127.28.25.2). New cstate: current_term: 1 leader_uuid: "e150f1d004b543b192ff403e505878a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e150f1d004b543b192ff403e505878a6" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 41063 } health_report { overall_health: HEALTHY } } }
I20260811 11:20:00.489187 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.486345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.28.25.2:41063)
0811 11:20:00.486396 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:00.486398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:00.486410 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:20:00.486460 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:00.486461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:00.486491 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:00.486571 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:00.486831 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.487443 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.488311 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.488520 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.488605 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.488788 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:00.488882 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:00.489003 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.489059 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.489072 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:20:00.491294 16793 raft_consensus.cc:1261] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7 [term 1 FOLLOWER]: Refusing update from remote peer e150f1d004b543b192ff403e505878a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:00.491303 17084 raft_consensus.cc:1261] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15 [term 1 FOLLOWER]: Refusing update from remote peer e150f1d004b543b192ff403e505878a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:00.491672 17168 consensus_queue.cc:1035] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c79a74a7b54854a4d0598eeed28be7" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:00.491838 17170 consensus_queue.cc:1035] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e11f4d622bed433998589cfec6d25c15" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:00.494412 17174 mvcc.cc:205] Tried to move back new op lower bound from 7317287733209014272 to 7317287733090979840. Current Snapshot: MvccSnapshot[applied={T|T < 7317287733209014272}]
I20260811 11:20:00.495359 17175 mvcc.cc:205] Tried to move back new op lower bound from 7317287733209014272 to 7317287733090979840. Current Snapshot: MvccSnapshot[applied={T|T < 7317287733209014272}]
I20260811 11:20:00.497264 16633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:20:00.497350 16633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:20:00.498129 16633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 458ae28622ca4d67bdef71361150d327: Sending DeleteTablet for 3 replicas of tablet ea1fcd924f4548c795f4e38f71ea820f
I20260811 11:20:00.498505 17063 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1fcd924f4548c795f4e38f71ea820f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:00 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:34074
I20260811 11:20:00.498533 16771 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1fcd924f4548c795f4e38f71ea820f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:00 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:36980
I20260811 11:20:00.498656 16918 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1fcd924f4548c795f4e38f71ea820f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:00 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:39790
I20260811 11:20:00.498754 17185 tablet_replica.cc:330] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: stopping tablet replica
I20260811 11:20:00.498857 17185 raft_consensus.cc:2229] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:20:00.498977 17185 raft_consensus.cc:2258] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:20:00.499104 17186 tablet_replica.cc:330] T ea1fcd924f4548c795f4e38f71ea820f P 11c79a74a7b54854a4d0598eeed28be7: stopping tablet replica
I20260811 11:20:00.499176 17187 tablet_replica.cc:330] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6: stopping tablet replica
I20260811 11:20:00.499266 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 16702
I20260811 11:20:00.499289 17187 raft_consensus.cc:2229] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:20:00.499462 17185 ts_tablet_manager.cc:1905] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:20:00.503742 17185 ts_tablet_manager.cc:1918] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:20:00.503834 17185 log.cc:1196] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/ea1fcd924f4548c795f4e38f71ea820f
I20260811 11:20:00.504096 17185 ts_tablet_manager.cc:1939] T ea1fcd924f4548c795f4e38f71ea820f P e11f4d622bed433998589cfec6d25c15: Deleting consensus metadata
I20260811 11:20:00.504664 16608 catalog_manager.cc:4841] TS e11f4d622bed433998589cfec6d25c15 (127.28.25.3:40299): tablet ea1fcd924f4548c795f4e38f71ea820f (table test-table [id=9553ba0ddd48427ea535329c4199170b]) successfully deleted
I20260811 11:20:00.507670 17187 raft_consensus.cc:2258] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:20:00.508204 17187 ts_tablet_manager.cc:1905] T ea1fcd924f4548c795f4e38f71ea820f P e150f1d004b543b192ff403e505878a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260811 11:20:00.509090 16608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:36655 (error 108)
I20260811 11:20:00.509292 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 16848
I20260811 11:20:00.509811 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.509685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:36655 (local address 127.0.0.1:36990)
0811 11:20:00.509738 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:00.509770 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:36655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260811 11:20:00.509970 16608 catalog_manager.cc:4584] TS 11c79a74a7b54854a4d0598eeed28be7 (127.28.25.1:36655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea1fcd924f4548c795f4e38f71ea820f: Network error: Client connection negotiation failed: client connection to 127.28.25.1:36655: connect: Connection refused (error 111)
I20260811 11:20:00.516695 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 16996
I20260811 11:20:00.516741 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.516599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:41063 (local address 127.0.0.1:39808)
0811 11:20:00.516663 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:00.516693 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.2:41063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 11:20:00.524613 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 16589
2026-08-11T11:20:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2380 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-11T11:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:00Z Disabled control of system clock
I20260811 11:20:00.557297 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:41469
--webserver_interface=127.28.25.62
--webserver_port=0
--builtin_ntp_servers=127.28.25.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:41469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:20:00.644747 17192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:00.645017 17192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:00.645041 17192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:00.646418 17192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:00.646489 17192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:00.646502 17192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:00.646515 17192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:00.646528 17192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:00.648046 17192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:41469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:41469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:00.648344 17192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:00.648675 17192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:00.654152 17199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:00.654219 17197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:00.654225 17192 server_base.cc:1034] running on GCE node
W20260811 11:20:00.654152 17202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:00.654649 17192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:00.655069 17192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:00.656265 17192 hybrid_clock.cc:648] HybridClock initialized: now 1786447200656249 us; error 34 us; skew 500 ppm
I20260811 11:20:00.657722 17192 webserver.cc:460] Webserver started at http://127.28.25.62:34851/ using document root <none> and password file <none>
I20260811 11:20:00.658004 17192 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:00.658082 17192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:00.658241 17192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:00.659520 17192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "891c097f864844319890a5c467a0ff1b"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "8483848d51fb7803e29fc85d62a9608b"
server_key_iv: "5402dab7afae11246f18894353ec2a41"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.659911 17192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "891c097f864844319890a5c467a0ff1b"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "8483848d51fb7803e29fc85d62a9608b"
server_key_iv: "5402dab7afae11246f18894353ec2a41"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.661302 17192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:20:00.662238 17210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:00.662467 17192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:00.662587 17192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "891c097f864844319890a5c467a0ff1b"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "8483848d51fb7803e29fc85d62a9608b"
server_key_iv: "5402dab7afae11246f18894353ec2a41"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.662667 17192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:00.685163 17192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:00.685552 17192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:00.685719 17192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:00.690289 17192 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:41469
I20260811 11:20:00.690333 17265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:41469 every 8 connection(s)
I20260811 11:20:00.690723 17192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:00.691865 17266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:00.692757 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 17192
I20260811 11:20:00.692914 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:00.693135 28772 external_mini_cluster.cc:1428] Setting key aea9aea77bd15229c8b5e27748834aa1
I20260811 11:20:00.695541 17266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b: Bootstrap starting.
I20260811 11:20:00.696312 17266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:00.696622 17266 log.cc:824] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:00.697554 17266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b: No bootstrap required, opened a new log
I20260811 11:20:00.699785 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.693746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.28.25.62:41469)
0811 11:20:00.693998 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:00.694008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:00.694042 (+ 34us) server_negotiation.cc:407] Connection header received
0811 11:20:00.694568 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:00.694577 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:00.694667 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:00.694846 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:00.695208 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.696566 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.697456 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.698057 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.698192 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.699023 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:00.699028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:00.699028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:00.699239 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:00.699284 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:00.699310 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:00.699379 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.699461 (+ 82us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.699553 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":106,"threads_started":1}
I20260811 11:20:00.700284 17266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891c097f864844319890a5c467a0ff1b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41469 } }
I20260811 11:20:00.700455 17266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:00.700495 17266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891c097f864844319890a5c467a0ff1b, State: Initialized, Role: FOLLOWER
I20260811 11:20:00.700610 17266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891c097f864844319890a5c467a0ff1b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41469 } }
I20260811 11:20:00.700690 17266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:00.700712 17266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:00.700780 17266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:00.701376 17266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891c097f864844319890a5c467a0ff1b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41469 } }
I20260811 11:20:00.701468 17266 leader_election.cc:302] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 891c097f864844319890a5c467a0ff1b; no voters:
I20260811 11:20:00.701669 17266 leader_election.cc:288] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:00.701771 17271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:00.701938 17271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [term 1 LEADER]: Becoming Leader. State: Replica: 891c097f864844319890a5c467a0ff1b, State: Running, Role: LEADER
I20260811 11:20:00.701956 17266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:00.702032 17271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891c097f864844319890a5c467a0ff1b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41469 } }
I20260811 11:20:00.702451 17272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "891c097f864844319890a5c467a0ff1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891c097f864844319890a5c467a0ff1b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41469 } } }
I20260811 11:20:00.702694 17272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:00.702622 17273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 891c097f864844319890a5c467a0ff1b. Latest consensus state: current_term: 1 leader_uuid: "891c097f864844319890a5c467a0ff1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891c097f864844319890a5c467a0ff1b" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41469 } } }
I20260811 11:20:00.702912 17273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:00.703202 17283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:00.703773 17283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:00.705498 17283 catalog_manager.cc:1347] Generated new cluster ID: a30612e187e7411b89926a3294bbd782
I20260811 11:20:00.705575 17283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:00.710598 17283 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:00.711387 17283 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:00.716049 17283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b: Generated new TSK 0
I20260811 11:20:00.716284 17283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:00.719791 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41469
--builtin_ntp_servers=127.28.25.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:20:00.808465 17293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:00.808686 17293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:00.808710 17293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:00.810151 17293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:00.810207 17293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:00.810252 17293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:20:00.812017 17293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:00.812319 17293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:00.812659 17293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:00.819226 17299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:00.819222 17300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:00.819217 17302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:00.819290 17293 server_base.cc:1034] running on GCE node
I20260811 11:20:00.819633 17293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:00.819969 17293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:00.821187 17293 hybrid_clock.cc:648] HybridClock initialized: now 1786447200821164 us; error 33 us; skew 500 ppm
I20260811 11:20:00.822687 17293 webserver.cc:460] Webserver started at http://127.28.25.1:34519/ using document root <none> and password file <none>
I20260811 11:20:00.823042 17293 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:00.823118 17293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:00.823282 17293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:00.824432 17293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "2c20cdf5cf00440fb4e24e514b96dbbd"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "618f030819ebc586d6480bb3cd1be1ea"
server_key_iv: "ab1ee89d22e7485def5ba030bffe2512"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.824858 17293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "2c20cdf5cf00440fb4e24e514b96dbbd"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "618f030819ebc586d6480bb3cd1be1ea"
server_key_iv: "ab1ee89d22e7485def5ba030bffe2512"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.826488 17293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:20:00.827647 17308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:00.828032 17293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:20:00.828145 17293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "2c20cdf5cf00440fb4e24e514b96dbbd"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "618f030819ebc586d6480bb3cd1be1ea"
server_key_iv: "ab1ee89d22e7485def5ba030bffe2512"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.828243 17293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:00.843384 17293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:00.843771 17293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:00.843945 17293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:00.844220 17293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:20:00.844605 17293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:20:00.844661 17293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:00.844712 17293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:20:00.844751 17293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:00.852353 17293 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:41971
I20260811 11:20:00.852494 17432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:41971 every 8 connection(s)
I20260811 11:20:00.852840 17293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:20:00.856256 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 17293
I20260811 11:20:00.856369 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:20:00.856616 28772 external_mini_cluster.cc:1428] Setting key 4ba5292233c1efacfc622199e731cbc0
I20260811 11:20:00.858114 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41469
--builtin_ntp_servers=127.28.25.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 11:20:00.860095 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.853907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:43391 (local address 127.28.25.62:41469)
0811 11:20:00.853997 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:00.854000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:00.854223 (+ 223us) server_negotiation.cc:407] Connection header received
0811 11:20:00.854862 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:00.854864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:00.854880 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:00.855028 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:00.855940 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.856621 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.858100 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.858418 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.858483 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.859687 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:00.859688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:00.859688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:00.859694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:00.859720 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:00.859724 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:00.859786 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:00.859899 (+ 113us) server_negotiation.cc:299] Negotiation successful
0811 11:20:00.859970 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 11:20:00.860330 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.853844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:41469 (local address 127.28.25.1:43391)
0811 11:20:00.854086 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:00.854126 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:00.854691 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:00.855142 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:00.855151 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:00.855161 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:00.855758 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.855764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.856794 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.856796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.857971 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:00.857975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:00.858518 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:00.858519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:00.858701 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:00.859492 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:20:00.859493 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:00.859539 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:20:00.859575 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:00.859804 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:00.859811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:00.859815 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:20:00.859896 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 11:20:00.860038 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":122,"threads_started":1}
I20260811 11:20:00.861068 17433 heartbeater.cc:347] Connected to a master server at 127.28.25.62:41469
I20260811 11:20:00.861171 17433 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:00.861405 17433 heartbeater.cc:511] Master 127.28.25.62:41469 requested a full tablet report, sending...
I20260811 11:20:00.862093 17229 ts_manager.cc:194] Registered new tserver with Master: 2c20cdf5cf00440fb4e24e514b96dbbd (127.28.25.1:41971)
I20260811 11:20:00.863125 17229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.1:43391
W20260811 11:20:00.948067 17436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:00.948302 17436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:00.948328 17436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:00.949488 17436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:00.949524 17436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:00.949550 17436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:20:00.950922 17436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:00.951211 17436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:00.951498 17436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:00.956905 17441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:00.956905 17442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:00.956956 17445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:00.957197 17436 server_base.cc:1034] running on GCE node
I20260811 11:20:00.957477 17436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:00.957779 17436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:00.959000 17436 hybrid_clock.cc:648] HybridClock initialized: now 1786447200958942 us; error 31 us; skew 500 ppm
I20260811 11:20:00.960371 17436 webserver.cc:460] Webserver started at http://127.28.25.2:39693/ using document root <none> and password file <none>
I20260811 11:20:00.960640 17436 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:00.960717 17436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:00.960875 17436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:00.961992 17436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "cbb52c6a02f24cd89a5211f68a7b72db"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "85b933afc12a5d4a43293024778dd77b"
server_key_iv: "2f9c6037d0f4104e7d1765b1d95ce1e3"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.962378 17436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "cbb52c6a02f24cd89a5211f68a7b72db"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "85b933afc12a5d4a43293024778dd77b"
server_key_iv: "2f9c6037d0f4104e7d1765b1d95ce1e3"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.964061 17436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:20:00.965165 17451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:00.965477 17436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:00.965576 17436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "cbb52c6a02f24cd89a5211f68a7b72db"
format_stamp: "Formatted at 2026-08-11 11:20:00 on dist-test-slave-2w3w"
server_key: "85b933afc12a5d4a43293024778dd77b"
server_key_iv: "2f9c6037d0f4104e7d1765b1d95ce1e3"
server_key_version: "encryptionkey@0"
I20260811 11:20:00.965696 17436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:00.990000 17436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:00.990408 17436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:00.990597 17436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:00.990880 17436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:20:00.991325 17436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:20:00.991391 17436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:00.991441 17436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:20:00.991470 17436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:00.998492 17436 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:43839
I20260811 11:20:00.998538 17581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:43839 every 8 connection(s)
I20260811 11:20:00.998948 17436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:20:01.005146 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.999984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:56917 (local address 127.28.25.62:41469)
0811 11:20:01.000052 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:01.000053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:01.000370 (+ 317us) server_negotiation.cc:407] Connection header received
0811 11:20:01.000914 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:01.000916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:01.000927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:01.000986 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:01.001661 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.002195 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.003489 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.003691 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.003825 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.004795 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:01.004796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:01.004797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:01.004801 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:01.004816 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:01.004820 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:01.004870 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:01.005033 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 11:20:01.005068 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 11:20:01.005406 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:00.999950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:41469 (local address 127.28.25.2:56917)
0811 11:20:01.000184 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:01.000258 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:01.000801 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:01.001044 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:01.001048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:01.001057 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:01.001576 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.001580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.002347 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.002348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.003383 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.003387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.003846 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.003847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.003977 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.004590 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:20:01.004590 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:01.004621 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:20:01.004647 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:01.004904 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:01.004911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:01.004914 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:20:01.005006 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 11:20:01.005099 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":119,"threads_started":1}
I20260811 11:20:01.006067 17582 heartbeater.cc:347] Connected to a master server at 127.28.25.62:41469
I20260811 11:20:01.006184 17582 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:01.006384 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 17436
I20260811 11:20:01.006414 17582 heartbeater.cc:511] Master 127.28.25.62:41469 requested a full tablet report, sending...
I20260811 11:20:01.006498 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:20:01.006743 28772 external_mini_cluster.cc:1428] Setting key af931985eb00776069031a0e5da7fd51
I20260811 11:20:01.006922 17229 ts_manager.cc:194] Registered new tserver with Master: cbb52c6a02f24cd89a5211f68a7b72db (127.28.25.2:43839)
I20260811 11:20:01.007592 17229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.2:56917
I20260811 11:20:01.007983 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41469
--builtin_ntp_servers=127.28.25.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:20:01.102378 17585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:01.102655 17585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:01.102684 17585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:01.104063 17585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:01.104110 17585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:01.104138 17585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:20:01.105589 17585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:41469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:01.105928 17585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:01.106235 17585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:01.111943 17590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:01.111951 17593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:01.111977 17591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:01.112181 17585 server_base.cc:1034] running on GCE node
I20260811 11:20:01.112437 17585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:01.112709 17585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:01.113888 17585 hybrid_clock.cc:648] HybridClock initialized: now 1786447201113884 us; error 33 us; skew 500 ppm
I20260811 11:20:01.115447 17585 webserver.cc:460] Webserver started at http://127.28.25.3:40535/ using document root <none> and password file <none>
I20260811 11:20:01.115738 17585 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:01.115815 17585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:01.115978 17585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:01.117154 17585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "81a7107e8e154fa7926bac90557f4726"
format_stamp: "Formatted at 2026-08-11 11:20:01 on dist-test-slave-2w3w"
server_key: "68d31350c9a264a5c55594909555b2ec"
server_key_iv: "a61e4f1e72c10816d8f3676be09bb1ea"
server_key_version: "encryptionkey@0"
I20260811 11:20:01.117547 17585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "81a7107e8e154fa7926bac90557f4726"
format_stamp: "Formatted at 2026-08-11 11:20:01 on dist-test-slave-2w3w"
server_key: "68d31350c9a264a5c55594909555b2ec"
server_key_iv: "a61e4f1e72c10816d8f3676be09bb1ea"
server_key_version: "encryptionkey@0"
I20260811 11:20:01.119199 17585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260811 11:20:01.120174 17600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:01.120471 17585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:01.120589 17585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "81a7107e8e154fa7926bac90557f4726"
format_stamp: "Formatted at 2026-08-11 11:20:01 on dist-test-slave-2w3w"
server_key: "68d31350c9a264a5c55594909555b2ec"
server_key_iv: "a61e4f1e72c10816d8f3676be09bb1ea"
server_key_version: "encryptionkey@0"
I20260811 11:20:01.120672 17585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:01.150309 17585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:01.150693 17585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:01.150964 17585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:01.151289 17585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:20:01.151656 17585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:20:01.151715 17585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:01.151760 17585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:20:01.151788 17585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:01.158169 17585 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:39037
I20260811 11:20:01.158275 17723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:39037 every 8 connection(s)
I20260811 11:20:01.158666 17585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:20:01.165051 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 17585
I20260811 11:20:01.165191 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:20:01.165199 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.159654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:39427 (local address 127.28.25.62:41469)
0811 11:20:01.159759 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:01.159762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:01.160159 (+ 397us) server_negotiation.cc:407] Connection header received
0811 11:20:01.160538 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:01.160540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:01.160560 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:01.160647 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:01.161480 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.162009 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.163493 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.163686 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.163759 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.164778 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:01.164780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:01.164780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:01.164787 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:01.164811 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:01.164818 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:01.164935 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:01.165059 (+ 124us) server_negotiation.cc:299] Negotiation successful
0811 11:20:01.165098 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 11:20:01.165428 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.159590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:41469 (local address 127.28.25.3:39427)
0811 11:20:01.159838 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:01.159882 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:01.160436 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:01.160824 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:01.160828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:01.160837 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:01.161385 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.161390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.162248 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.162249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.163403 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.163407 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.163790 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.163791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.163978 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.164589 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:20:01.164589 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:01.164623 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:20:01.164648 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:01.164950 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:01.164957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:01.164961 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:20:01.165031 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 11:20:01.165148 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":132,"threads_started":1}
I20260811 11:20:01.165473 28772 external_mini_cluster.cc:1428] Setting key 42f9397ae3884e8fef7fbebabf7f98c6
I20260811 11:20:01.165978 17724 heartbeater.cc:347] Connected to a master server at 127.28.25.62:41469
I20260811 11:20:01.166105 17724 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:01.166337 17724 heartbeater.cc:511] Master 127.28.25.62:41469 requested a full tablet report, sending...
I20260811 11:20:01.166757 17228 ts_manager.cc:194] Registered new tserver with Master: 81a7107e8e154fa7926bac90557f4726 (127.28.25.3:39037)
I20260811 11:20:01.167214 17228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.3:39427
I20260811 11:20:01.177942 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:20:01.183513 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.179826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.28.25.62:41469)
0811 11:20:01.179880 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:01.179882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:01.180017 (+ 135us) server_negotiation.cc:407] Connection header received
0811 11:20:01.180064 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:01.180064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:01.180074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:01.180131 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:01.180304 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.180801 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.181741 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.181989 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.182062 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.183101 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:01.183102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:01.183102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:01.183109 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:01.183125 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:01.183130 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:01.183226 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:01.183426 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 11:20:01.183442 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 11:20:01.185830 17228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:20:01.187849 17228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:20:01.194916 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.190469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:43839 (local address 127.0.0.1:38038)
0811 11:20:01.190867 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:01.190913 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:01.191156 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:01.191477 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:01.191478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:01.191481 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:01.191534 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.191536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.192465 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.192467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.193793 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.193796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.194583 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.194584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.194664 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.194678 (+ 14us) client_negotiation.cc:769] Sending connection context
0811 11:20:01.194759 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 11:20:01.194824 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260811 11:20:01.194926 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.190674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.28.25.2:43839)
0811 11:20:01.191287 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:01.191291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:01.191301 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:20:01.191359 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:01.191361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:01.191376 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:01.191613 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:01.191666 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.192372 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.193903 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.194477 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.194583 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.194692 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:01.194793 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 11:20:01.194835 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":165,"threads_started":1}
I20260811 11:20:01.195230 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.190469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:39037 (local address 127.0.0.1:45102)
0811 11:20:01.190942 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:01.190952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:01.191063 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:01.191361 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:01.191362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:01.191364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:01.191433 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.191435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.192843 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.192844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.193769 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.193771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.194898 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.194899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.194953 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.194992 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 11:20:01.195076 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 11:20:01.195123 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":12,"thread_start_us":65,"threads_started":1}
I20260811 11:20:01.195276 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.190658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.28.25.3:39037)
0811 11:20:01.190857 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:01.190862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:01.191062 (+ 200us) server_negotiation.cc:407] Connection header received
0811 11:20:01.191173 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:01.191176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:01.191195 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:01.191281 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:01.191527 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.192725 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.193893 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.194671 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.194767 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.194864 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:01.195127 (+ 263us) server_negotiation.cc:299] Negotiation successful
0811 11:20:01.195184 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260811 11:20:01.195230 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.190475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:41971 (local address 127.0.0.1:46060)
0811 11:20:01.191034 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:01.191042 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:01.191093 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:01.191326 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:01.191330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:01.191333 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:01.191433 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.191436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.192825 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.192826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.193892 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.193893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.194800 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.194801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.194877 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.194883 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:20:01.194916 (+ 33us) client_negotiation.cc:240] Negotiation successful
0811 11:20:01.194988 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":7,"spinlock_wait_cycles":896,"thread_start_us":144,"threads_started":1}
I20260811 11:20:01.195878 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.190771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.28.25.1:41971)
0811 11:20:01.190938 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:01.190941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:01.191107 (+ 166us) server_negotiation.cc:407] Connection header received
0811 11:20:01.191180 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:01.191182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:01.191196 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:01.191521 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:01.191591 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.192734 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.193968 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.194678 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.195578 (+ 900us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.195688 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:01.195717 (+ 29us) server_negotiation.cc:299] Negotiation successful
0811 11:20:01.195768 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260811 11:20:01.196066 17656 tablet_service.cc:1467] Processing CreateTablet for tablet f7370aa980ea4764b5c08aec193ecea7 (DEFAULT_TABLE table=test-table [id=51477448de5d41e89d40d01c60d7edac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:01.196070 17508 tablet_service.cc:1467] Processing CreateTablet for tablet f7370aa980ea4764b5c08aec193ecea7 (DEFAULT_TABLE table=test-table [id=51477448de5d41e89d40d01c60d7edac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:01.196491 17656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7370aa980ea4764b5c08aec193ecea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:01.196491 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7370aa980ea4764b5c08aec193ecea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:01.196527 17361 tablet_service.cc:1467] Processing CreateTablet for tablet f7370aa980ea4764b5c08aec193ecea7 (DEFAULT_TABLE table=test-table [id=51477448de5d41e89d40d01c60d7edac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:01.196934 17361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7370aa980ea4764b5c08aec193ecea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:01.199958 17739 tablet_bootstrap.cc:492] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd: Bootstrap starting.
I20260811 11:20:01.200417 17741 tablet_bootstrap.cc:492] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: Bootstrap starting.
I20260811 11:20:01.200843 17739 tablet_bootstrap.cc:654] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:01.201041 17741 tablet_bootstrap.cc:654] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:01.201205 17739 log.cc:824] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:01.201390 17741 log.cc:824] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:01.201789 17740 tablet_bootstrap.cc:492] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: Bootstrap starting.
I20260811 11:20:01.202266 17739 tablet_bootstrap.cc:492] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd: No bootstrap required, opened a new log
I20260811 11:20:01.202345 17739 ts_tablet_manager.cc:1397] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:20:01.202394 17741 tablet_bootstrap.cc:492] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: No bootstrap required, opened a new log
I20260811 11:20:01.202462 17741 ts_tablet_manager.cc:1397] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:20:01.202715 17740 tablet_bootstrap.cc:654] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:01.203146 17740 log.cc:824] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:01.204036 17740 tablet_bootstrap.cc:492] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: No bootstrap required, opened a new log
I20260811 11:20:01.204118 17740 ts_tablet_manager.cc:1397] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:20:01.204983 17741 raft_consensus.cc:348] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43839 } } peers { permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 } } peers { permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 } }
I20260811 11:20:01.205153 17741 raft_consensus.cc:374] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:01.205199 17741 raft_consensus.cc:729] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb52c6a02f24cd89a5211f68a7b72db, State: Initialized, Role: FOLLOWER
I20260811 11:20:01.205333 17741 consensus_queue.cc:260] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43839 } } peers { permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 } } peers { permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 } }
I20260811 11:20:01.205451 17739 raft_consensus.cc:348] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43839 } } peers { permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 } } peers { permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 } }
I20260811 11:20:01.205621 17741 ts_tablet_manager.cc:1428] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 11:20:01.205633 17739 raft_consensus.cc:374] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:01.205662 17582 heartbeater.cc:503] Master 127.28.25.62:41469 was elected leader, sending a full tablet report...
I20260811 11:20:01.205675 17739 raft_consensus.cc:729] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c20cdf5cf00440fb4e24e514b96dbbd, State: Initialized, Role: FOLLOWER
I20260811 11:20:01.205812 17739 consensus_queue.cc:260] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43839 } } peers { permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 } } peers { permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 } }
I20260811 11:20:01.206101 17739 ts_tablet_manager.cc:1428] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 11:20:01.206153 17433 heartbeater.cc:503] Master 127.28.25.62:41469 was elected leader, sending a full tablet report...
I20260811 11:20:01.207445 17740 raft_consensus.cc:348] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43839 } } peers { permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 } } peers { permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 } }
I20260811 11:20:01.207578 17740 raft_consensus.cc:374] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:01.207599 17740 raft_consensus.cc:729] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a7107e8e154fa7926bac90557f4726, State: Initialized, Role: FOLLOWER
I20260811 11:20:01.207710 17740 consensus_queue.cc:260] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43839 } } peers { permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 } } peers { permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 } }
I20260811 11:20:01.207932 17740 ts_tablet_manager.cc:1428] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260811 11:20:01.208046 17724 heartbeater.cc:503] Master 127.28.25.62:41469 was elected leader, sending a full tablet report...
W20260811 11:20:01.249672 17583 tablet.cc:2369] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:20:01.353744 17434 tablet.cc:2369] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:20:01.409380 17725 tablet.cc:2369] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:20:01.432260 17745 raft_consensus.cc:482] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:20:01.432451 17745 raft_consensus.cc:504] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43839 } } peers { permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 } } peers { permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 } }
I20260811 11:20:01.432866 17745 leader_election.cc:288] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81a7107e8e154fa7926bac90557f4726 (127.28.25.3:39037), 2c20cdf5cf00440fb4e24e514b96dbbd (127.28.25.1:41971)
I20260811 11:20:01.435915 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.433081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:41971 (local address 127.28.25.2:49455)
0811 11:20:01.433143 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:01.433156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:01.433228 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:01.433413 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:01.433414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:01.433415 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:01.433493 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.433495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.434205 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.434206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.435085 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.435087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.435637 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.435638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.435724 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.435727 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:20:01.435792 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:20:01.435837 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 11:20:01.435915 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.433186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:49455 (local address 127.28.25.1:41971)
0811 11:20:01.433261 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:01.433263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:01.433278 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:20:01.433312 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:01.433313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:01.433324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:01.433396 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:01.433567 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.434110 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.435175 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.435542 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.435634 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.435645 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:01.435814 (+ 169us) server_negotiation.cc:299] Negotiation successful
0811 11:20:01.435852 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:20:01.436062 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.433099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:39037 (local address 127.28.25.2:36727)
0811 11:20:01.433329 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:01.433339 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:01.433436 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:01.433595 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:01.433596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:01.433597 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:01.433678 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.433679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.434372 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.434373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.435169 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:01.435171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.435757 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.435758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:01.435845 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.435848 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:20:01.435913 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:20:01.435955 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":144,"threads_started":1}
I20260811 11:20:01.436261 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.433271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:36727 (local address 127.28.25.3:39037)
0811 11:20:01.433322 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:01.433325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:01.433423 (+ 98us) server_negotiation.cc:407] Connection header received
0811 11:20:01.433516 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:01.433518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:01.433526 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:01.433588 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:01.433808 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.434277 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.435275 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.435647 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.435750 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.435760 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:01.435931 (+ 171us) server_negotiation.cc:299] Negotiation successful
0811 11:20:01.436186 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:20:01.436347 17381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7370aa980ea4764b5c08aec193ecea7" candidate_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" is_pre_election: true
I20260811 11:20:01.436529 17381 raft_consensus.cc:2454] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbb52c6a02f24cd89a5211f68a7b72db in term 0.
I20260811 11:20:01.436563 17676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7370aa980ea4764b5c08aec193ecea7" candidate_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a7107e8e154fa7926bac90557f4726" is_pre_election: true
I20260811 11:20:01.436743 17676 raft_consensus.cc:2454] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbb52c6a02f24cd89a5211f68a7b72db in term 0.
I20260811 11:20:01.436832 17456 leader_election.cc:302] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c20cdf5cf00440fb4e24e514b96dbbd, cbb52c6a02f24cd89a5211f68a7b72db; no voters:
I20260811 11:20:01.437034 17745 raft_consensus.cc:2781] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:20:01.437079 17745 raft_consensus.cc:482] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:20:01.437095 17745 raft_consensus.cc:3037] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:01.437840 17745 raft_consensus.cc:504] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43839 } } peers { permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 } } peers { permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 } }
I20260811 11:20:01.437995 17745 leader_election.cc:288] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [CANDIDATE]: Term 1 election: Requested vote from peers 81a7107e8e154fa7926bac90557f4726 (127.28.25.3:39037), 2c20cdf5cf00440fb4e24e514b96dbbd (127.28.25.1:41971)
I20260811 11:20:01.438210 17381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7370aa980ea4764b5c08aec193ecea7" candidate_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd"
I20260811 11:20:01.438210 17676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7370aa980ea4764b5c08aec193ecea7" candidate_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a7107e8e154fa7926bac90557f4726"
I20260811 11:20:01.438301 17676 raft_consensus.cc:3037] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:01.438301 17381 raft_consensus.cc:3037] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:01.439306 17676 raft_consensus.cc:2454] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb52c6a02f24cd89a5211f68a7b72db in term 1.
I20260811 11:20:01.439306 17381 raft_consensus.cc:2454] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb52c6a02f24cd89a5211f68a7b72db in term 1.
I20260811 11:20:01.439545 17456 leader_election.cc:302] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c20cdf5cf00440fb4e24e514b96dbbd, cbb52c6a02f24cd89a5211f68a7b72db; no voters:
I20260811 11:20:01.439680 17745 raft_consensus.cc:2781] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:01.439874 17745 raft_consensus.cc:686] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 1 LEADER]: Becoming Leader. State: Replica: cbb52c6a02f24cd89a5211f68a7b72db, State: Running, Role: LEADER
I20260811 11:20:01.439999 17745 consensus_queue.cc:237] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43839 } } peers { permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 } } peers { permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 } }
I20260811 11:20:01.440943 17229 catalog_manager.cc:5495] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db reported cstate change: term changed from 0 to 1, leader changed from <none> to cbb52c6a02f24cd89a5211f68a7b72db (127.28.25.2). New cstate: current_term: 1 leader_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb52c6a02f24cd89a5211f68a7b72db" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 43839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 } health_report { overall_health: UNKNOWN } } }
I20260811 11:20:01.480805 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.478108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.28.25.2:43839)
0811 11:20:01.478213 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:01.478215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:01.478225 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:20:01.478269 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:01.478270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:01.478296 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:01.478377 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:01.478498 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.479119 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.479934 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:01.480125 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:01.480187 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:01.480425 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:01.480521 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:01.480608 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:01.480661 (+ 53us) server_negotiation.cc:299] Negotiation successful
0811 11:20:01.480673 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 11:20:01.482990 17381 raft_consensus.cc:1261] T f7370aa980ea4764b5c08aec193ecea7 P 2c20cdf5cf00440fb4e24e514b96dbbd [term 1 FOLLOWER]: Refusing update from remote peer cbb52c6a02f24cd89a5211f68a7b72db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:01.483069 17676 raft_consensus.cc:1261] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726 [term 1 FOLLOWER]: Refusing update from remote peer cbb52c6a02f24cd89a5211f68a7b72db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:01.483407 17749 consensus_queue.cc:1035] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c20cdf5cf00440fb4e24e514b96dbbd" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:01.483620 17745 consensus_queue.cc:1035] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a7107e8e154fa7926bac90557f4726" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 39037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:01.486229 17752 mvcc.cc:205] Tried to move back new op lower bound from 7317287737270878208 to 7317287737098883072. Current Snapshot: MvccSnapshot[applied={T|T < 7317287737270878208}]
I20260811 11:20:01.486788 17754 mvcc.cc:205] Tried to move back new op lower bound from 7317287737270878208 to 7317287737098883072. Current Snapshot: MvccSnapshot[applied={T|T < 7317287737270878208}]
I20260811 11:20:01.487323 17753 mvcc.cc:205] Tried to move back new op lower bound from 7317287737270878208 to 7317287737098883072. Current Snapshot: MvccSnapshot[applied={T|T < 7317287737270878208}]
I20260811 11:20:01.489413 17228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:20:01.489521 17228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:20:01.490432 17228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 891c097f864844319890a5c467a0ff1b: Sending DeleteTablet for 3 replicas of tablet f7370aa980ea4764b5c08aec193ecea7
I20260811 11:20:01.490850 17656 tablet_service.cc:1514] Processing DeleteTablet for tablet f7370aa980ea4764b5c08aec193ecea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:01 UTC) from {username='slave'} at 127.0.0.1:45102
I20260811 11:20:01.490868 17508 tablet_service.cc:1514] Processing DeleteTablet for tablet f7370aa980ea4764b5c08aec193ecea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:01 UTC) from {username='slave'} at 127.0.0.1:38038
I20260811 11:20:01.490991 17361 tablet_service.cc:1514] Processing DeleteTablet for tablet f7370aa980ea4764b5c08aec193ecea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:01 UTC) from {username='slave'} at 127.0.0.1:46060
I20260811 11:20:01.491186 17764 tablet_replica.cc:330] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: stopping tablet replica
I20260811 11:20:01.491215 17763 tablet_replica.cc:330] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: stopping tablet replica
I20260811 11:20:01.491324 17763 raft_consensus.cc:2229] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:20:01.491343 17764 raft_consensus.cc:2229] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:20:01.491458 17763 raft_consensus.cc:2258] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:20:01.491614 17764 raft_consensus.cc:2258] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:20:01.491659 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 17293
I20260811 11:20:01.491940 17763 ts_tablet_manager.cc:1905] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:20:01.494223 17764 ts_tablet_manager.cc:1905] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:20:01.494308 17763 ts_tablet_manager.cc:1918] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:20:01.494390 17763 log.cc:1196] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/f7370aa980ea4764b5c08aec193ecea7
I20260811 11:20:01.494670 17763 ts_tablet_manager.cc:1939] T f7370aa980ea4764b5c08aec193ecea7 P cbb52c6a02f24cd89a5211f68a7b72db: Deleting consensus metadata
I20260811 11:20:01.495167 17215 catalog_manager.cc:4841] TS cbb52c6a02f24cd89a5211f68a7b72db (127.28.25.2:43839): tablet f7370aa980ea4764b5c08aec193ecea7 (table test-table [id=51477448de5d41e89d40d01c60d7edac]) successfully deleted
I20260811 11:20:01.496507 17764 ts_tablet_manager.cc:1918] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:20:01.496675 17764 log.cc:1196] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/f7370aa980ea4764b5c08aec193ecea7
I20260811 11:20:01.497056 17764 ts_tablet_manager.cc:1939] T f7370aa980ea4764b5c08aec193ecea7 P 81a7107e8e154fa7926bac90557f4726: Deleting consensus metadata
I20260811 11:20:01.497989 17212 catalog_manager.cc:4841] TS 81a7107e8e154fa7926bac90557f4726 (127.28.25.3:39037): tablet f7370aa980ea4764b5c08aec193ecea7 (table test-table [id=51477448de5d41e89d40d01c60d7edac]) successfully deleted
W20260811 11:20:01.503414 17216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:41971 (error 108)
I20260811 11:20:01.504066 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:01.503907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:41971 (local address 127.0.0.1:46076)
0811 11:20:01.503972 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:01.504026 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:41971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 11:20:01.504132 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 17436
W20260811 11:20:01.507490 17216 catalog_manager.cc:4584] TS 2c20cdf5cf00440fb4e24e514b96dbbd (127.28.25.1:41971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7370aa980ea4764b5c08aec193ecea7: Network error: Client connection negotiation failed: client connection to 127.28.25.1:41971: connect: Connection refused (error 111)
I20260811 11:20:01.513062 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 17585
I20260811 11:20:01.520836 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 17192
2026-08-11T11:20:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (994 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:01 dist-test-slave-2w3w krb5kdc[17769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:01 dist-test-slave-2w3w krb5kdc[17769](info): set up 2 sockets
Aug 11 11:20:01 dist-test-slave-2w3w krb5kdc[17769](info): commencing operation
krb5kdc: starting...
W20260811 11:20:02.721609 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:02 dist-test-slave-2w3w krb5kdc[17769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:20:02.863511 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:44141
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:44141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:02.949950 17784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:02.950237 17784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:02.950284 17784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:02.951627 17784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:02.951718 17784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:02.951743 17784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:02.951776 17784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:02.951803 17784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:02.953526 17784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:44141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:44141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:02.953863 17784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:02.954134 17784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:02.959801 17790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:02.959884 17789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:02.959797 17794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:02.960219 17784 server_base.cc:1034] running on GCE node
I20260811 11:20:02.960502 17784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:02.960845 17784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:02.962049 17784 hybrid_clock.cc:648] HybridClock initialized: now 1786447202962030 us; error 46 us; skew 500 ppm
Aug 11 11:20:02 dist-test-slave-2w3w krb5kdc[17769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447202, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:02.965143 17784 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:20:02.965713 17784 webserver.cc:460] Webserver started at http://127.28.25.62:46603/ using document root <none> and password file <none>
I20260811 11:20:02.965982 17784 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:02.966055 17784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:02.966355 17784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:02.968071 17784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "0d46da29670746f5a810a8a8925444f0"
format_stamp: "Formatted at 2026-08-11 11:20:02 on dist-test-slave-2w3w"
server_key: "17cc729bfe60f1b4fc55d34753c775fc"
server_key_iv: "02938f1cfc8dbf44e3ed85e32adda689"
server_key_version: "encryptionkey@0"
I20260811 11:20:02.968502 17784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "0d46da29670746f5a810a8a8925444f0"
format_stamp: "Formatted at 2026-08-11 11:20:02 on dist-test-slave-2w3w"
server_key: "17cc729bfe60f1b4fc55d34753c775fc"
server_key_iv: "02938f1cfc8dbf44e3ed85e32adda689"
server_key_version: "encryptionkey@0"
I20260811 11:20:02.970070 17784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:20:02.971207 17803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:02.971494 17784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 11:20:02.971621 17784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "0d46da29670746f5a810a8a8925444f0"
format_stamp: "Formatted at 2026-08-11 11:20:02 on dist-test-slave-2w3w"
server_key: "17cc729bfe60f1b4fc55d34753c775fc"
server_key_iv: "02938f1cfc8dbf44e3ed85e32adda689"
server_key_version: "encryptionkey@0"
I20260811 11:20:02.971710 17784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:03.000183 17784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:03.002651 17784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:03.002857 17784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:03.007439 17784 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:44141
I20260811 11:20:03.007506 17858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:44141 every 8 connection(s)
I20260811 11:20:03.007893 17784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:03.008841 17859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:03.010437 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 17784
I20260811 11:20:03.010564 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:03.010819 28772 external_mini_cluster.cc:1428] Setting key 3de658b1d44adb9ed67ff96d79ed5fd6
I20260811 11:20:03.012079 17859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0: Bootstrap starting.
I20260811 11:20:03.012850 17859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:03.013227 17859 log.cc:824] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:03.014223 17859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0: No bootstrap required, opened a new log
I20260811 11:20:03.016768 17859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d46da29670746f5a810a8a8925444f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44141 } }
I20260811 11:20:03.017000 17859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:03.017026 17859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d46da29670746f5a810a8a8925444f0, State: Initialized, Role: FOLLOWER
I20260811 11:20:03.017179 17859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d46da29670746f5a810a8a8925444f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44141 } }
I20260811 11:20:03.017280 17859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:03.017331 17859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:03.017409 17859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:03.018375 17859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d46da29670746f5a810a8a8925444f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44141 } }
I20260811 11:20:03.018522 17859 leader_election.cc:302] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d46da29670746f5a810a8a8925444f0; no voters:
I20260811 11:20:03.018762 17859 leader_election.cc:288] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:03.018846 17864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:03.019109 17859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:03.019147 17864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [term 1 LEADER]: Becoming Leader. State: Replica: 0d46da29670746f5a810a8a8925444f0, State: Running, Role: LEADER
I20260811 11:20:03.019260 17864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d46da29670746f5a810a8a8925444f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44141 } }
Aug 11 11:20:03 dist-test-slave-2w3w krb5kdc[17769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:03.019994 17865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d46da29670746f5a810a8a8925444f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d46da29670746f5a810a8a8925444f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44141 } } }
I20260811 11:20:03.020107 17865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:03.020341 17874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:03.020011 17866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d46da29670746f5a810a8a8925444f0. Latest consensus state: current_term: 1 leader_uuid: "0d46da29670746f5a810a8a8925444f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d46da29670746f5a810a8a8925444f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44141 } } }
I20260811 11:20:03.020607 17866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:03.021517 17874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:03.023620 17874 catalog_manager.cc:1347] Generated new cluster ID: 6f73a21f8c4840c888f9a7b16e828eef
I20260811 11:20:03.023681 17874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:03.024658 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.011614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.28.25.62:44141)
0811 11:20:03.011811 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.011821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.011859 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:20:03.012354 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.012361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.012421 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.012548 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:03.013477 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.014916 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.015800 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.016572 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.016712 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.020099 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:03.020105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:03.020105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:03.020119 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:03.022608 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:03.023085 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:03.023088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:03.023089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:03.023210 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:03.023487 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:03.023488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:03.023489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:03.023834 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:03.023985 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.024132 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.024276 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260811 11:20:03.034672 17874 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:03.035422 17874 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:03.042332 17874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0: Generated new TSK 0
I20260811 11:20:03.042622 17874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.25.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
Entry for principal kudu/127.28.25.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab.
I20260811 11:20:03.082198 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44141
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:03.170142 17886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:03.170374 17886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:03.170403 17886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:03.171674 17886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:03.171721 17886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:03.171751 17886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:20:03.173285 17886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.1.keytab
--principal=kudu/127.28.25.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:03.173561 17886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:03.173820 17886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:03.179340 17891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:03.179421 17892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:03.179428 17894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:03.179601 17886 server_base.cc:1034] running on GCE node
I20260811 11:20:03.179845 17886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:03.180159 17886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:03.181334 17886 hybrid_clock.cc:648] HybridClock initialized: now 1786447203181313 us; error 41 us; skew 500 ppm
Aug 11 11:20:03 dist-test-slave-2w3w krb5kdc[17769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447203, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:03.184142 17886 init.cc:377] Logged in from keytab as kudu/127.28.25.1@KRBTEST.COM (short username kudu)
I20260811 11:20:03.184629 17886 webserver.cc:460] Webserver started at http://127.28.25.1:46191/ using document root <none> and password file <none>
I20260811 11:20:03.184827 17886 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:03.184868 17886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:03.184962 17886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:03.186020 17886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "5b42bce760814e7e97269b67b2cd0775"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "e1cc6b91e7202c1ea123f78a8dee4cb4"
server_key_iv: "9b14f36fef4d2bdfdaf8cc21a6ce6195"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.186343 17886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "5b42bce760814e7e97269b67b2cd0775"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "e1cc6b91e7202c1ea123f78a8dee4cb4"
server_key_iv: "9b14f36fef4d2bdfdaf8cc21a6ce6195"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.187813 17886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:20:03.188580 17901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:03.188815 17886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:03.188900 17886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "5b42bce760814e7e97269b67b2cd0775"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "e1cc6b91e7202c1ea123f78a8dee4cb4"
server_key_iv: "9b14f36fef4d2bdfdaf8cc21a6ce6195"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.188962 17886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:03.212122 17886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:03.214219 17886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:03.214341 17886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:03.214540 17886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:20:03.214887 17886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:20:03.214922 17886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:03.214946 17886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:20:03.215027 17886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:03.221362 17886 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:45199
I20260811 11:20:03.221506 18023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:45199 every 8 connection(s)
I20260811 11:20:03.221813 17886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
Aug 11 11:20:03 dist-test-slave-2w3w krb5kdc[17769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447203, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.1@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:03.228797 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 17886
I20260811 11:20:03.228935 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:20:03.229169 28772 external_mini_cluster.cc:1428] Setting key cbe641bbcd0a06348b09dda0a7c4669e
I20260811 11:20:03.232316 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.222676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:35253 (local address 127.28.25.62:44141)
0811 11:20:03.222787 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.222790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.223018 (+ 228us) server_negotiation.cc:407] Connection header received
0811 11:20:03.223585 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.223587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.223605 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.223673 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:03.224735 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.225346 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.226341 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.226541 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.226601 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.229036 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:03.229037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:03.229038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:03.229047 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:03.230553 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:03.231131 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:03.231132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:03.231133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:03.231200 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:03.231524 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:03.231525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:03.231525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:03.231655 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:03.231759 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.232100 (+ 341us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.232214 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 11:20:03.232568 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.222595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:44141 (local address 127.28.25.1:35253)
0811 11:20:03.222828 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:03.222874 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:03.223464 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:03.223787 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:03.223790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:03.224416 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:03.224626 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.224630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.225446 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.225447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.226255 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.226258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.226621 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.226622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.226733 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.227345 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:03.227354 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:03.228875 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:03.230639 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:03.230646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:03.230649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:03.231006 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:03.231281 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:03.231282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:03.231283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:03.231441 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:03.231765 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:03.231774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:03.231939 (+ 165us) client_negotiation.cc:769] Sending connection context
0811 11:20:03.232058 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 11:20:03.232288 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260811 11:20:03.233233 18024 heartbeater.cc:347] Connected to a master server at 127.28.25.62:44141
I20260811 11:20:03.233348 18024 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:03.233557 18024 heartbeater.cc:511] Master 127.28.25.62:44141 requested a full tablet report, sending...
I20260811 11:20:03.234186 17821 ts_manager.cc:194] Registered new tserver with Master: 5b42bce760814e7e97269b67b2cd0775 (127.28.25.1:45199)
I20260811 11:20:03.235276 17821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.1@KRBTEST.COM'} at 127.28.25.1:35253
WARNING: no policy specified for kudu/127.28.25.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
Entry for principal kudu/127.28.25.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab.
I20260811 11:20:03.258752 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44141
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:03.344374 18030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:03.344604 18030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:03.344632 18030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:03.345803 18030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:03.345845 18030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:03.345876 18030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:20:03.347395 18030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.2.keytab
--principal=kudu/127.28.25.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:03.347630 18030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:03.347826 18030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:03.353340 18036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:03.353415 18030 server_base.cc:1034] running on GCE node
W20260811 11:20:03.353569 18039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:03.353374 18035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:03.353845 18030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:03.354136 18030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:03.355311 18030 hybrid_clock.cc:648] HybridClock initialized: now 1786447203355305 us; error 37 us; skew 500 ppm
Aug 11 11:20:03 dist-test-slave-2w3w krb5kdc[17769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447203, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:03.358192 18030 init.cc:377] Logged in from keytab as kudu/127.28.25.2@KRBTEST.COM (short username kudu)
I20260811 11:20:03.358706 18030 webserver.cc:460] Webserver started at http://127.28.25.2:39539/ using document root <none> and password file <none>
I20260811 11:20:03.359032 18030 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:03.359107 18030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:03.359285 18030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:03.360489 18030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "467c340ed79c4360aec8ea1a10d4acfb"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "4d3472ff2cf0e149155024606ea12237"
server_key_iv: "3d18049036bfa88a8a15d49cf152bd23"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.360886 18030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "467c340ed79c4360aec8ea1a10d4acfb"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "4d3472ff2cf0e149155024606ea12237"
server_key_iv: "3d18049036bfa88a8a15d49cf152bd23"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.363057 18030 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 11:20:03.364048 18047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:03.364253 18030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:03.364375 18030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "467c340ed79c4360aec8ea1a10d4acfb"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "4d3472ff2cf0e149155024606ea12237"
server_key_iv: "3d18049036bfa88a8a15d49cf152bd23"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.364481 18030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:03.376791 18030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:03.379364 18030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:03.379534 18030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:03.379805 18030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:20:03.380162 18030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:20:03.380198 18030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:03.380223 18030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:20:03.380264 18030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:03.386835 18030 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:45259
I20260811 11:20:03.386915 18173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:45259 every 8 connection(s)
I20260811 11:20:03.387360 18030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
Aug 11 11:20:03 dist-test-slave-2w3w krb5kdc[17769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447203, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.2@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:03.395560 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 18030
I20260811 11:20:03.395716 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:20:03.395958 28772 external_mini_cluster.cc:1428] Setting key 671e58d506dacb633f7a0e4a448b081d
I20260811 11:20:03.398856 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.388374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:53153 (local address 127.28.25.62:44141)
0811 11:20:03.388502 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.388505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.388693 (+ 188us) server_negotiation.cc:407] Connection header received
0811 11:20:03.389409 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.389411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.389424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.389470 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:03.390574 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.391174 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.392331 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.392547 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.392631 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.395192 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:03.395194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:03.395194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:03.395202 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:03.397178 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:03.397696 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:03.397698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:03.397698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:03.397766 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:03.398062 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:03.398063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:03.398063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:03.398199 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:03.398285 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.398601 (+ 316us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.398734 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 11:20:03.399165 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.388256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:44141 (local address 127.28.25.2:53153)
0811 11:20:03.388521 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:03.388581 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:03.389252 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:03.389546 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:03.389553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:03.390220 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:03.390449 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.390455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.391282 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.391283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.392230 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.392233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.392644 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.392645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.392791 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.393428 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:03.393437 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:03.395079 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:03.397283 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:03.397296 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:03.397307 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:03.397594 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:03.397845 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:03.397846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:03.397847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:03.397986 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:03.398274 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:03.398283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:03.398455 (+ 172us) client_negotiation.cc:769] Sending connection context
0811 11:20:03.398580 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 11:20:03.398834 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":116,"threads_started":1}
I20260811 11:20:03.399792 18174 heartbeater.cc:347] Connected to a master server at 127.28.25.62:44141
I20260811 11:20:03.399945 18174 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:03.400170 18174 heartbeater.cc:511] Master 127.28.25.62:44141 requested a full tablet report, sending...
I20260811 11:20:03.400671 17821 ts_manager.cc:194] Registered new tserver with Master: 467c340ed79c4360aec8ea1a10d4acfb (127.28.25.2:45259)
I20260811 11:20:03.401108 17821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.2@KRBTEST.COM'} at 127.28.25.2:53153
WARNING: no policy specified for kudu/127.28.25.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
Entry for principal kudu/127.28.25.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab.
I20260811 11:20:03.430755 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44141
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:03.520509 18179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:03.520752 18179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:03.520781 18179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:03.522336 18179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:03.522406 18179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:03.522440 18179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:20:03.524580 18179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.3.keytab
--principal=kudu/127.28.25.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:44141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:03.524855 18179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:03.525172 18179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:03.530889 18184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:03.530937 18188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:03.531001 18185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:03.531219 18179 server_base.cc:1034] running on GCE node
I20260811 11:20:03.531569 18179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:03.531921 18179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:03.533103 18179 hybrid_clock.cc:648] HybridClock initialized: now 1786447203533082 us; error 50 us; skew 500 ppm
Aug 11 11:20:03 dist-test-slave-2w3w krb5kdc[17769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447203, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:03.536232 18179 init.cc:377] Logged in from keytab as kudu/127.28.25.3@KRBTEST.COM (short username kudu)
I20260811 11:20:03.536816 18179 webserver.cc:460] Webserver started at http://127.28.25.3:43491/ using document root <none> and password file <none>
I20260811 11:20:03.537074 18179 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:03.537149 18179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:03.537319 18179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:03.538697 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "7314ee09e7ce445b87e0e54eb4e4699e"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "85f38788f1a684dd5924cc9eda2a3960"
server_key_iv: "58d5c29fd9db03f8081bce19cd31b6aa"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.539227 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "7314ee09e7ce445b87e0e54eb4e4699e"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "85f38788f1a684dd5924cc9eda2a3960"
server_key_iv: "58d5c29fd9db03f8081bce19cd31b6aa"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.542054 18179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260811 11:20:03.543452 18195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:03.543689 18179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:03.543865 18179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "7314ee09e7ce445b87e0e54eb4e4699e"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "85f38788f1a684dd5924cc9eda2a3960"
server_key_iv: "58d5c29fd9db03f8081bce19cd31b6aa"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.543979 18179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:03.584596 18179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:03.586843 18179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:03.587076 18179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:03.587354 18179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:20:03.587755 18179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:20:03.587824 18179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:03.587867 18179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:20:03.587895 18179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:03.594391 18179 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:38173
I20260811 11:20:03.594491 18318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:38173 every 8 connection(s)
I20260811 11:20:03.594861 18179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:20:03.597994 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 18179
I20260811 11:20:03.598093 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:20:03.598304 28772 external_mini_cluster.cc:1428] Setting key afd9ada2db8caef7730ee6b4f000134a
Aug 11 11:20:03 dist-test-slave-2w3w krb5kdc[17769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447203, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.3@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:03.607236 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.595684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:44141 (local address 127.28.25.3:47519)
0811 11:20:03.595857 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:03.595898 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:03.596385 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:03.596640 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:03.596644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:03.597270 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:03.597500 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.597506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.598461 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.598461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.600046 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.600051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.600649 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.600651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.600833 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.601457 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:03.601467 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:03.603290 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:03.605363 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:03.605387 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:03.605389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:03.605678 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:03.606041 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:03.606042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:03.606042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:03.606181 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:03.606531 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:03.606538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:03.606690 (+ 152us) client_negotiation.cc:769] Sending connection context
0811 11:20:03.606789 (+ 99us) client_negotiation.cc:240] Negotiation successful
0811 11:20:03.607025 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260811 11:20:03.607270 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.595715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:47519 (local address 127.28.25.62:44141)
0811 11:20:03.595767 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.595769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.595996 (+ 227us) server_negotiation.cc:407] Connection header received
0811 11:20:03.596504 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.596505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.596517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.596575 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:03.597603 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.598305 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.600196 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.600517 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.600595 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.603406 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:03.603407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:03.603408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:03.603416 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:03.605252 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:03.605773 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:03.605774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:03.605774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:03.605926 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:03.606302 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:03.606303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:03.606303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:03.606444 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:03.606509 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.607092 (+ 583us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.607196 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 11:20:03.607913 18319 heartbeater.cc:347] Connected to a master server at 127.28.25.62:44141
I20260811 11:20:03.608017 18319 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:03.608183 18319 heartbeater.cc:511] Master 127.28.25.62:44141 requested a full tablet report, sending...
I20260811 11:20:03.608536 17821 ts_manager.cc:194] Registered new tserver with Master: 7314ee09e7ce445b87e0e54eb4e4699e (127.28.25.3:38173)
I20260811 11:20:03.608959 17821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.25.3@KRBTEST.COM'} at 127.28.25.3:47519
I20260811 11:20:03.611675 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:20:03.620834 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.613170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.28.25.62:44141)
0811 11:20:03.613214 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.613218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.613359 (+ 141us) server_negotiation.cc:407] Connection header received
0811 11:20:03.613421 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.613421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.613430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.613482 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:03.614065 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.614557 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.615489 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.615716 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.615830 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.617429 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:03.617430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:03.617431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:03.617438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:03.619343 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:03.619818 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:03.619819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:03.619819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:03.619882 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:03.620185 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:03.620186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:03.620186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:03.620401 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:03.620488 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.620622 (+ 134us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.620704 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 11:20:03.623221 17821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:20:03.624935 17821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:20:03.632153 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.627984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55958 (local address 127.28.25.3:38173)
0811 11:20:03.628229 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.628235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.628248 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:20:03.628295 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.628298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.628316 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.628427 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:03.628621 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.629489 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.630864 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.631664 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.631755 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.631837 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.632003 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.632051 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260811 11:20:03.632139 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.627740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:38173 (local address 127.0.0.1:55958)
0811 11:20:03.627961 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:03.628012 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:03.628090 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:03.628433 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:03.628438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:03.628440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:03.628525 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.628527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.629607 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.629608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.630744 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.630747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.631757 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.631758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.631853 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.631870 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 11:20:03.631980 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 11:20:03.632051 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":55,"threads_started":1}
I20260811 11:20:03.632781 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.627739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:45259 (local address 127.0.0.1:57314)
0811 11:20:03.628222 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:03.628233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:03.628275 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:03.628623 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:03.628624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:03.628626 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:03.628696 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.628698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.630202 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.630203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.631371 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.631374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.632494 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.632495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.632578 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.632584 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:20:03.632638 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 11:20:03.632688 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":3,"spinlock_wait_cycles":384,"thread_start_us":133,"threads_started":1}
I20260811 11:20:03.632907 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.627750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:45199 (local address 127.0.0.1:32918)
0811 11:20:03.628402 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:03.628412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:03.628458 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:03.628783 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:03.628783 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:03.628785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:03.628821 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.628823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.630246 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.630247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.631393 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.631396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.632624 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.632625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.632697 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.632702 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:20:03.632765 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 11:20:03.632808 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":122,"threads_started":1}
I20260811 11:20:03.633276 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.628077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.28.25.1:45199)
0811 11:20:03.628300 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.628304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.628550 (+ 246us) server_negotiation.cc:407] Connection header received
0811 11:20:03.628600 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.628602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.628619 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.628697 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:03.628913 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.630119 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.631526 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.632526 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.632608 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.632667 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.632808 (+ 141us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.633188 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":123,"threads_started":1}
I20260811 11:20:03.633312 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.627994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.28.25.2:45259)
0811 11:20:03.628362 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.628367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.628379 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:20:03.628429 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.628431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.628448 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.628530 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:03.628784 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.630068 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.631473 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.632292 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.632386 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.632451 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.633167 (+ 716us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.633222 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":96,"threads_started":1}
I20260811 11:20:03.633203 18244 tablet_service.cc:1467] Processing CreateTablet for tablet 80cf36c12479442d828b386605a672ce (DEFAULT_TABLE table=test-table [id=c1069250f2f6419293f2dff8a861c4fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:03.633626 18244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80cf36c12479442d828b386605a672ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:03.634184 17955 tablet_service.cc:1467] Processing CreateTablet for tablet 80cf36c12479442d828b386605a672ce (DEFAULT_TABLE table=test-table [id=c1069250f2f6419293f2dff8a861c4fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:03.634548 17955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80cf36c12479442d828b386605a672ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:03.635663 18102 tablet_service.cc:1467] Processing CreateTablet for tablet 80cf36c12479442d828b386605a672ce (DEFAULT_TABLE table=test-table [id=c1069250f2f6419293f2dff8a861c4fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:03.636040 18102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80cf36c12479442d828b386605a672ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:03.637733 18338 tablet_bootstrap.cc:492] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: Bootstrap starting.
I20260811 11:20:03.638876 18338 tablet_bootstrap.cc:654] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:03.639665 18338 log.cc:824] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:03.639734 18340 tablet_bootstrap.cc:492] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: Bootstrap starting.
I20260811 11:20:03.640594 18340 tablet_bootstrap.cc:654] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:03.640628 18338 tablet_bootstrap.cc:492] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: No bootstrap required, opened a new log
I20260811 11:20:03.640698 18338 ts_tablet_manager.cc:1397] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:20:03.641247 18340 log.cc:824] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:03.641691 18339 tablet_bootstrap.cc:492] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775: Bootstrap starting.
I20260811 11:20:03.642504 18339 tablet_bootstrap.cc:654] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:03.643218 18339 log.cc:824] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:03.643369 18340 tablet_bootstrap.cc:492] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: No bootstrap required, opened a new log
I20260811 11:20:03.643469 18340 ts_tablet_manager.cc:1397] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 11:20:03.643467 18338 raft_consensus.cc:348] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 } } peers { permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 } } peers { permanent_uuid: "5b42bce760814e7e97269b67b2cd0775" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 45199 } }
I20260811 11:20:03.643708 18338 raft_consensus.cc:374] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:03.643762 18338 raft_consensus.cc:729] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7314ee09e7ce445b87e0e54eb4e4699e, State: Initialized, Role: FOLLOWER
I20260811 11:20:03.643949 18338 consensus_queue.cc:260] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 } } peers { permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 } } peers { permanent_uuid: "5b42bce760814e7e97269b67b2cd0775" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 45199 } }
I20260811 11:20:03.644289 18338 ts_tablet_manager.cc:1428] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 11:20:03.644357 18339 tablet_bootstrap.cc:492] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775: No bootstrap required, opened a new log
I20260811 11:20:03.644424 18339 ts_tablet_manager.cc:1397] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 11:20:03.644438 18319 heartbeater.cc:503] Master 127.28.25.62:44141 was elected leader, sending a full tablet report...
I20260811 11:20:03.645715 18340 raft_consensus.cc:348] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 } } peers { permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 } } peers { permanent_uuid: "5b42bce760814e7e97269b67b2cd0775" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 45199 } }
I20260811 11:20:03.645921 18340 raft_consensus.cc:374] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:03.645973 18340 raft_consensus.cc:729] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467c340ed79c4360aec8ea1a10d4acfb, State: Initialized, Role: FOLLOWER
I20260811 11:20:03.646090 18340 consensus_queue.cc:260] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 } } peers { permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 } } peers { permanent_uuid: "5b42bce760814e7e97269b67b2cd0775" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 45199 } }
I20260811 11:20:03.646430 18340 ts_tablet_manager.cc:1428] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 11:20:03.646512 18174 heartbeater.cc:503] Master 127.28.25.62:44141 was elected leader, sending a full tablet report...
I20260811 11:20:03.646907 18339 raft_consensus.cc:348] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 } } peers { permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 } } peers { permanent_uuid: "5b42bce760814e7e97269b67b2cd0775" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 45199 } }
I20260811 11:20:03.647136 18339 raft_consensus.cc:374] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:03.647197 18339 raft_consensus.cc:729] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b42bce760814e7e97269b67b2cd0775, State: Initialized, Role: FOLLOWER
I20260811 11:20:03.647439 18339 consensus_queue.cc:260] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 } } peers { permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 } } peers { permanent_uuid: "5b42bce760814e7e97269b67b2cd0775" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 45199 } }
I20260811 11:20:03.647603 18347 raft_consensus.cc:482] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:20:03.647770 18339 ts_tablet_manager.cc:1428] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260811 11:20:03.647744 18347 raft_consensus.cc:504] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 } } peers { permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 } } peers { permanent_uuid: "5b42bce760814e7e97269b67b2cd0775" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 45199 } }
I20260811 11:20:03.647918 18024 heartbeater.cc:503] Master 127.28.25.62:44141 was elected leader, sending a full tablet report...
I20260811 11:20:03.648850 18347 leader_election.cc:288] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 467c340ed79c4360aec8ea1a10d4acfb (127.28.25.2:45259), 7314ee09e7ce445b87e0e54eb4e4699e (127.28.25.3:38173)
I20260811 11:20:03.652174 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.648290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:38173 (local address 127.28.25.1:46509)
0811 11:20:03.648426 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:03.648439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:03.648489 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:03.648701 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:03.648702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:03.648704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:03.648779 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.648781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.649609 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.649610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.651091 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.651094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.651938 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.651939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.652013 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.652017 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:20:03.652067 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 11:20:03.652101 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260811 11:20:03.652189 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.648374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:46509 (local address 127.28.25.3:38173)
0811 11:20:03.648427 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.648429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.648551 (+ 122us) server_negotiation.cc:407] Connection header received
0811 11:20:03.648581 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.648582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.648595 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.648648 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:03.648864 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.649547 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.651205 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.651831 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.651908 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.651978 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.652083 (+ 105us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.652121 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 11:20:03.652647 18267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cf36c12479442d828b386605a672ce" candidate_uuid: "5b42bce760814e7e97269b67b2cd0775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" is_pre_election: true
I20260811 11:20:03.652812 18267 raft_consensus.cc:2454] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b42bce760814e7e97269b67b2cd0775 in term 0.
I20260811 11:20:03.653018 17902 leader_election.cc:302] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b42bce760814e7e97269b67b2cd0775, 7314ee09e7ce445b87e0e54eb4e4699e; no voters:
I20260811 11:20:03.653187 18348 raft_consensus.cc:2781] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:20:03.653282 18348 raft_consensus.cc:482] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:20:03.653311 18348 raft_consensus.cc:3037] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:03.653395 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.648299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:45259 (local address 127.28.25.1:46819)
0811 11:20:03.648468 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:03.648479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:03.648547 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:03.648678 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:03.648679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:03.648681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:03.648858 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.648861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.651206 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.651207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.652319 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:03.652322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.653109 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.653110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:03.653201 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.653207 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:20:03.653270 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 11:20:03.653318 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":20,"thread_start_us":86,"threads_started":1}
I20260811 11:20:03.653452 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.648337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:46819 (local address 127.28.25.2:45259)
0811 11:20:03.648379 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.648381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.648538 (+ 157us) server_negotiation.cc:407] Connection header received
0811 11:20:03.648601 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.648603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.648614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.648672 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:03.648958 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.651095 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.652410 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.653005 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.653107 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.653119 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.653307 (+ 188us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.653355 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 11:20:03.653798 18123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cf36c12479442d828b386605a672ce" candidate_uuid: "5b42bce760814e7e97269b67b2cd0775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "467c340ed79c4360aec8ea1a10d4acfb" is_pre_election: true
I20260811 11:20:03.653950 18123 raft_consensus.cc:2454] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b42bce760814e7e97269b67b2cd0775 in term 0.
I20260811 11:20:03.654261 18348 raft_consensus.cc:504] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 } } peers { permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 } } peers { permanent_uuid: "5b42bce760814e7e97269b67b2cd0775" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 45199 } }
I20260811 11:20:03.654414 18348 leader_election.cc:288] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [CANDIDATE]: Term 1 election: Requested vote from peers 467c340ed79c4360aec8ea1a10d4acfb (127.28.25.2:45259), 7314ee09e7ce445b87e0e54eb4e4699e (127.28.25.3:38173)
I20260811 11:20:03.654565 18123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cf36c12479442d828b386605a672ce" candidate_uuid: "5b42bce760814e7e97269b67b2cd0775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "467c340ed79c4360aec8ea1a10d4acfb"
I20260811 11:20:03.654567 18267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cf36c12479442d828b386605a672ce" candidate_uuid: "5b42bce760814e7e97269b67b2cd0775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7314ee09e7ce445b87e0e54eb4e4699e"
I20260811 11:20:03.654645 18123 raft_consensus.cc:3037] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:03.654647 18267 raft_consensus.cc:3037] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:03.655647 18123 raft_consensus.cc:2454] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b42bce760814e7e97269b67b2cd0775 in term 1.
I20260811 11:20:03.655659 18267 raft_consensus.cc:2454] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b42bce760814e7e97269b67b2cd0775 in term 1.
I20260811 11:20:03.655829 17903 leader_election.cc:302] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 467c340ed79c4360aec8ea1a10d4acfb, 5b42bce760814e7e97269b67b2cd0775; no voters:
I20260811 11:20:03.655951 18348 raft_consensus.cc:2781] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:03.656037 18348 raft_consensus.cc:686] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [term 1 LEADER]: Becoming Leader. State: Replica: 5b42bce760814e7e97269b67b2cd0775, State: Running, Role: LEADER
I20260811 11:20:03.656127 18348 consensus_queue.cc:237] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 } } peers { permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 } } peers { permanent_uuid: "5b42bce760814e7e97269b67b2cd0775" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 45199 } }
I20260811 11:20:03.657070 17821 catalog_manager.cc:5495] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b42bce760814e7e97269b67b2cd0775 (127.28.25.1). New cstate: current_term: 1 leader_uuid: "5b42bce760814e7e97269b67b2cd0775" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b42bce760814e7e97269b67b2cd0775" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 45199 } health_report { overall_health: HEALTHY } } }
I20260811 11:20:03.668648 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.665826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.28.25.1:45199)
0811 11:20:03.665905 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.665907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.665944 (+ 37us) server_negotiation.cc:407] Connection header received
0811 11:20:03.666070 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.666071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.666095 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.666179 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:03.666311 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.666794 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.667666 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.667869 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.667946 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.668137 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:03.668269 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:03.668343 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.668525 (+ 182us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.668554 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:20:03.670828 18267 raft_consensus.cc:1261] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e [term 1 FOLLOWER]: Refusing update from remote peer 5b42bce760814e7e97269b67b2cd0775: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:03.670821 18123 raft_consensus.cc:1261] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb [term 1 FOLLOWER]: Refusing update from remote peer 5b42bce760814e7e97269b67b2cd0775: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:03.671275 18347 consensus_queue.cc:1035] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7314ee09e7ce445b87e0e54eb4e4699e" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:03.671425 18348 consensus_queue.cc:1035] T 80cf36c12479442d828b386605a672ce P 5b42bce760814e7e97269b67b2cd0775 [LEADER]: Connected to new peer: Peer: permanent_uuid: "467c340ed79c4360aec8ea1a10d4acfb" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:03.675565 18354 mvcc.cc:205] Tried to move back new op lower bound from 7317287746232205312 to 7317287746176270336. Current Snapshot: MvccSnapshot[applied={T|T < 7317287746232205312}]
I20260811 11:20:03.677305 17821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:20:03.677433 17821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:20:03.678330 17821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d46da29670746f5a810a8a8925444f0: Sending DeleteTablet for 3 replicas of tablet 80cf36c12479442d828b386605a672ce
I20260811 11:20:03.678754 17955 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cf36c12479442d828b386605a672ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:03 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:32918
I20260811 11:20:03.678859 18102 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cf36c12479442d828b386605a672ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:03 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:57314
I20260811 11:20:03.679215 18244 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cf36c12479442d828b386605a672ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:03 UTC) from {username='kudu', principal='kudu/127.28.25.62@KRBTEST.COM'} at 127.0.0.1:55958
I20260811 11:20:03.679406 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 17886
I20260811 11:20:03.679472 18364 tablet_replica.cc:330] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: stopping tablet replica
I20260811 11:20:03.680083 18365 tablet_replica.cc:330] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: stopping tablet replica
I20260811 11:20:03.680249 18365 raft_consensus.cc:2229] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:20:03.680408 18365 raft_consensus.cc:2258] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:20:03.687156 18364 raft_consensus.cc:2229] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:20:03.687388 18364 raft_consensus.cc:2258] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:20:03.687990 18364 ts_tablet_manager.cc:1905] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:20:03.689641 18364 ts_tablet_manager.cc:1918] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:20:03.689728 18364 log.cc:1196] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/80cf36c12479442d828b386605a672ce
I20260811 11:20:03.690016 18364 ts_tablet_manager.cc:1939] T 80cf36c12479442d828b386605a672ce P 467c340ed79c4360aec8ea1a10d4acfb: Deleting consensus metadata
I20260811 11:20:03.690503 17805 catalog_manager.cc:4841] TS 467c340ed79c4360aec8ea1a10d4acfb (127.28.25.2:45259): tablet 80cf36c12479442d828b386605a672ce (table test-table [id=c1069250f2f6419293f2dff8a861c4fb]) successfully deleted
I20260811 11:20:03.691272 18365 ts_tablet_manager.cc:1905] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:20:03.692795 18365 ts_tablet_manager.cc:1918] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:20:03.692965 18365 log.cc:1196] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/80cf36c12479442d828b386605a672ce
I20260811 11:20:03.693243 18365 ts_tablet_manager.cc:1939] T 80cf36c12479442d828b386605a672ce P 7314ee09e7ce445b87e0e54eb4e4699e: Deleting consensus metadata
I20260811 11:20:03.693740 17804 catalog_manager.cc:4841] TS 7314ee09e7ce445b87e0e54eb4e4699e (127.28.25.3:38173): tablet 80cf36c12479442d828b386605a672ce (table test-table [id=c1069250f2f6419293f2dff8a861c4fb]) successfully deleted
W20260811 11:20:03.694922 17807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:45199 (error 108)
I20260811 11:20:03.695133 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 18030
I20260811 11:20:03.696660 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.696465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:45199 (local address 127.0.0.1:32932)
0811 11:20:03.696541 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:03.696609 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260811 11:20:03.696849 17807 catalog_manager.cc:4584] TS 5b42bce760814e7e97269b67b2cd0775 (127.28.25.1:45199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80cf36c12479442d828b386605a672ce: Network error: Client connection negotiation failed: client connection to 127.28.25.1:45199: connect: Connection refused (error 111)
I20260811 11:20:03.703545 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 18179
I20260811 11:20:03.713581 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 17784
2026-08-11T11:20:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2195 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-11T11:20:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:03Z Disabled control of system clock
I20260811 11:20:03.748051 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33109
--webserver_interface=127.28.25.62
--webserver_port=0
--builtin_ntp_servers=127.28.25.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:33109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:20:03.833206 18371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:03.833461 18371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:03.833492 18371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:03.834937 18371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:03.835057 18371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:03.835074 18371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:03.835088 18371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:03.835101 18371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:03.836512 18371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:33109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:33109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:03.836753 18371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:03.837023 18371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:03.842442 18380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:03.842442 18376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:03.842489 18378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:03.842772 18371 server_base.cc:1034] running on GCE node
I20260811 11:20:03.843163 18371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:03.843458 18371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:03.844626 18371 hybrid_clock.cc:648] HybridClock initialized: now 1786447203844609 us; error 30 us; skew 500 ppm
I20260811 11:20:03.846143 18371 webserver.cc:460] Webserver started at http://127.28.25.62:44881/ using document root <none> and password file <none>
I20260811 11:20:03.846377 18371 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:03.846424 18371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:03.846522 18371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:03.847805 18371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "4d2784714a3642d4a9d967b1f042c6e5"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "cc3cfed3ab1cf91822b5f7a72fc75483"
server_key_iv: "49120f6488eb4be70ab22b5534c86454"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.848140 18371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "4d2784714a3642d4a9d967b1f042c6e5"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "cc3cfed3ab1cf91822b5f7a72fc75483"
server_key_iv: "49120f6488eb4be70ab22b5534c86454"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.849571 18371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:20:03.850534 18387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:03.850773 18371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:03.850872 18371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "4d2784714a3642d4a9d967b1f042c6e5"
format_stamp: "Formatted at 2026-08-11 11:20:03 on dist-test-slave-2w3w"
server_key: "cc3cfed3ab1cf91822b5f7a72fc75483"
server_key_iv: "49120f6488eb4be70ab22b5534c86454"
server_key_version: "encryptionkey@0"
I20260811 11:20:03.850942 18371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:03.876509 18371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:03.876828 18371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:03.876945 18371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:03.881069 18371 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:33109
I20260811 11:20:03.881138 18446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:33109 every 8 connection(s)
I20260811 11:20:03.881561 18371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:03.882761 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:03.883960 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 18371
I20260811 11:20:03.884106 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:03.884354 28772 external_mini_cluster.cc:1428] Setting key e616d4f98136d332089fdd8d05ed7ea9
I20260811 11:20:03.886128 18448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5: Bootstrap starting.
I20260811 11:20:03.886821 18448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:03.887216 18448 log.cc:824] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:03.888329 18448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5: No bootstrap required, opened a new log
I20260811 11:20:03.890822 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:03.884968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.28.25.62:33109)
0811 11:20:03.885228 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:03.885253 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:03.885291 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:20:03.885945 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:03.885953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:03.886022 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:03.886184 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:03.886424 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.887495 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.888735 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:03.889175 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:03.889283 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:03.890083 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:03.890087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:03.890088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:03.890259 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:03.890292 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:03.890304 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:03.890358 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:03.890498 (+ 140us) server_negotiation.cc:299] Negotiation successful
0811 11:20:03.890571 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":118,"threads_started":1}
I20260811 11:20:03.891080 18448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2784714a3642d4a9d967b1f042c6e5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33109 } }
I20260811 11:20:03.891261 18448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:03.891330 18448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2784714a3642d4a9d967b1f042c6e5, State: Initialized, Role: FOLLOWER
I20260811 11:20:03.891535 18448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2784714a3642d4a9d967b1f042c6e5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33109 } }
I20260811 11:20:03.891646 18448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:03.891697 18448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:03.891750 18448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:03.892424 18448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2784714a3642d4a9d967b1f042c6e5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33109 } }
I20260811 11:20:03.892550 18448 leader_election.cc:302] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d2784714a3642d4a9d967b1f042c6e5; no voters:
I20260811 11:20:03.892774 18448 leader_election.cc:288] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:03.892814 18454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:03.893026 18454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [term 1 LEADER]: Becoming Leader. State: Replica: 4d2784714a3642d4a9d967b1f042c6e5, State: Running, Role: LEADER
I20260811 11:20:03.893110 18448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:03.893126 18454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2784714a3642d4a9d967b1f042c6e5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33109 } }
I20260811 11:20:03.893492 18455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d2784714a3642d4a9d967b1f042c6e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2784714a3642d4a9d967b1f042c6e5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33109 } } }
I20260811 11:20:03.893595 18455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:03.893505 18456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d2784714a3642d4a9d967b1f042c6e5. Latest consensus state: current_term: 1 leader_uuid: "4d2784714a3642d4a9d967b1f042c6e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2784714a3642d4a9d967b1f042c6e5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 33109 } } }
I20260811 11:20:03.893745 18456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:03.893882 18461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:03.894603 18461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:03.896656 18461 catalog_manager.cc:1347] Generated new cluster ID: c71b14f2b290489c8c32eb16a3df6a69
I20260811 11:20:03.896718 18461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:03.913378 18461 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:03.914086 18461 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:03.925434 18461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5: Generated new TSK 0
I20260811 11:20:03.925699 18461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:03.931396 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.1:0
--local_ip_for_outbound_sockets=127.28.25.1
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33109
--builtin_ntp_servers=127.28.25.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:20:04.035137 18474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:04.035358 18474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:04.035384 18474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:04.036614 18474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:04.036687 18474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:04.036736 18474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.1
I20260811 11:20:04.038333 18474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.25.1
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.1
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:04.038630 18474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:04.039067 18474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:04.044723 18480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:04.044739 18479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:04.045145 18482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:04.046307 18474 server_base.cc:1034] running on GCE node
I20260811 11:20:04.046553 18474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:04.046855 18474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:04.048026 18474 hybrid_clock.cc:648] HybridClock initialized: now 1786447204048006 us; error 35 us; skew 500 ppm
I20260811 11:20:04.049665 18474 webserver.cc:460] Webserver started at http://127.28.25.1:37621/ using document root <none> and password file <none>
I20260811 11:20:04.049981 18474 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:04.050046 18474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:04.050189 18474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:04.051903 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/data/instance:
uuid: "f3046bafff234ecd931af83b28c31786"
format_stamp: "Formatted at 2026-08-11 11:20:04 on dist-test-slave-2w3w"
server_key: "f24e5584aa8af02d987aece0debb7e75"
server_key_iv: "95f92e33e04b8dccf857edb859f07878"
server_key_version: "encryptionkey@0"
I20260811 11:20:04.052404 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance:
uuid: "f3046bafff234ecd931af83b28c31786"
format_stamp: "Formatted at 2026-08-11 11:20:04 on dist-test-slave-2w3w"
server_key: "f24e5584aa8af02d987aece0debb7e75"
server_key_iv: "95f92e33e04b8dccf857edb859f07878"
server_key_version: "encryptionkey@0"
I20260811 11:20:04.054325 18474 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:20:04.055418 18488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:04.055732 18474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:20:04.055850 18474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/wal
uuid: "f3046bafff234ecd931af83b28c31786"
format_stamp: "Formatted at 2026-08-11 11:20:04 on dist-test-slave-2w3w"
server_key: "f24e5584aa8af02d987aece0debb7e75"
server_key_iv: "95f92e33e04b8dccf857edb859f07878"
server_key_version: "encryptionkey@0"
I20260811 11:20:04.055943 18474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:04.069418 18474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:04.069751 18474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:04.069876 18474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:04.070086 18474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:20:04.070480 18474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:20:04.070519 18474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:04.070544 18474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:20:04.070603 18474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:04.077721 18474 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.1:44999
I20260811 11:20:04.077805 18611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.1:44999 every 8 connection(s)
I20260811 11:20:04.078222 18474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/data/info.pb
I20260811 11:20:04.079177 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 18474
I20260811 11:20:04.079319 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-0/wal/instance
I20260811 11:20:04.079556 28772 external_mini_cluster.cc:1428] Setting key d8647fae80a0da07b250c6caf491545f
I20260811 11:20:04.080757 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.2:0
--local_ip_for_outbound_sockets=127.28.25.2
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33109
--builtin_ntp_servers=127.28.25.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 11:20:04.085362 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.079051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.1:54689 (local address 127.28.25.62:33109)
0811 11:20:04.079112 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:04.079114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:04.079448 (+ 334us) server_negotiation.cc:407] Connection header received
0811 11:20:04.080213 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:04.080215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:04.080260 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:04.080374 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:04.081255 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.082125 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.083708 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.083958 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.084030 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.085054 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:04.085055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:04.085056 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:04.085062 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:04.085079 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:04.085084 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:04.085129 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:04.085237 (+ 108us) server_negotiation.cc:299] Negotiation successful
0811 11:20:04.085263 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 11:20:04.085686 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.079055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33109 (local address 127.28.25.1:54689)
0811 11:20:04.079308 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:04.079356 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:04.080092 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:04.080474 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:04.080479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:04.080489 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:04.081143 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.081154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.082262 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.082263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.083606 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.083609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.084061 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.084062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.084230 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.084862 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:20:04.084862 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:04.084905 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:20:04.084938 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:04.085148 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:04.085154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:04.085157 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:20:04.085233 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 11:20:04.085367 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":117,"threads_started":1}
I20260811 11:20:04.086527 18612 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33109
I20260811 11:20:04.086652 18612 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:04.086894 18612 heartbeater.cc:511] Master 127.28.25.62:33109 requested a full tablet report, sending...
I20260811 11:20:04.087595 18406 ts_manager.cc:194] Registered new tserver with Master: f3046bafff234ecd931af83b28c31786 (127.28.25.1:44999)
I20260811 11:20:04.088675 18406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.1:54689
W20260811 11:20:04.171331 18615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:04.171566 18615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:04.171589 18615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:04.172945 18615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:04.173003 18615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:04.173031 18615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.2
I20260811 11:20:04.174433 18615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.25.2
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.2
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:04.174656 18615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:04.174997 18615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:04.180625 18624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:04.180835 18615 server_base.cc:1034] running on GCE node
W20260811 11:20:04.180629 18621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:04.180629 18622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:04.181255 18615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:04.181598 18615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:04.182799 18615 hybrid_clock.cc:648] HybridClock initialized: now 1786447204182777 us; error 41 us; skew 500 ppm
I20260811 11:20:04.184497 18615 webserver.cc:460] Webserver started at http://127.28.25.2:33641/ using document root <none> and password file <none>
I20260811 11:20:04.184854 18615 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:04.184947 18615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:04.185122 18615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:04.186443 18615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/data/instance:
uuid: "46013d595ff14c268435a975169273d3"
format_stamp: "Formatted at 2026-08-11 11:20:04 on dist-test-slave-2w3w"
server_key: "64396033749eebf1ddfbc3fde1dab8bd"
server_key_iv: "cbbb75eebf67d96f723b20f87ba91570"
server_key_version: "encryptionkey@0"
I20260811 11:20:04.186859 18615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance:
uuid: "46013d595ff14c268435a975169273d3"
format_stamp: "Formatted at 2026-08-11 11:20:04 on dist-test-slave-2w3w"
server_key: "64396033749eebf1ddfbc3fde1dab8bd"
server_key_iv: "cbbb75eebf67d96f723b20f87ba91570"
server_key_version: "encryptionkey@0"
I20260811 11:20:04.188516 18615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:04.189549 18630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:04.189778 18615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:04.189911 18615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/wal
uuid: "46013d595ff14c268435a975169273d3"
format_stamp: "Formatted at 2026-08-11 11:20:04 on dist-test-slave-2w3w"
server_key: "64396033749eebf1ddfbc3fde1dab8bd"
server_key_iv: "cbbb75eebf67d96f723b20f87ba91570"
server_key_version: "encryptionkey@0"
I20260811 11:20:04.190002 18615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:04.206940 18615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:04.207366 18615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:04.207547 18615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:04.207845 18615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:20:04.208232 18615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:20:04.208289 18615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:04.208334 18615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:20:04.208364 18615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:04.214684 18615 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.2:42455
I20260811 11:20:04.214778 18753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.2:42455 every 8 connection(s)
I20260811 11:20:04.215217 18615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/data/info.pb
I20260811 11:20:04.218327 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 18615
I20260811 11:20:04.218427 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/wal/instance
I20260811 11:20:04.218648 28772 external_mini_cluster.cc:1428] Setting key 4e134a195eb4c1dbf7d1e9d7cbf09297
I20260811 11:20:04.219905 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.25.3:0
--local_ip_for_outbound_sockets=127.28.25.3
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33109
--builtin_ntp_servers=127.28.25.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 11:20:04.221462 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.216155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:40357 (local address 127.28.25.62:33109)
0811 11:20:04.216221 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:04.216223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:04.216518 (+ 295us) server_negotiation.cc:407] Connection header received
0811 11:20:04.217079 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:04.217080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:04.217094 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:04.217160 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:04.217882 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.218425 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.219897 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.220158 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.220231 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.221166 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:04.221167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:04.221167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:04.221172 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:04.221188 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:04.221192 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:04.221245 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:04.221343 (+ 98us) server_negotiation.cc:299] Negotiation successful
0811 11:20:04.221374 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 11:20:04.221714 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.216136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33109 (local address 127.28.25.2:40357)
0811 11:20:04.216380 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:04.216417 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:04.216965 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:04.217241 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:04.217245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:04.217254 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:04.217775 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.217779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.218543 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.218544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.219793 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.219796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.220249 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.220250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.220390 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.220999 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:20:04.221000 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:04.221031 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:20:04.221055 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:04.221248 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:04.221254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:04.221257 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:20:04.221346 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 11:20:04.221458 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":111,"threads_started":1}
I20260811 11:20:04.222231 18755 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33109
I20260811 11:20:04.222368 18755 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:04.222609 18755 heartbeater.cc:511] Master 127.28.25.62:33109 requested a full tablet report, sending...
I20260811 11:20:04.223081 18406 ts_manager.cc:194] Registered new tserver with Master: 46013d595ff14c268435a975169273d3 (127.28.25.2:42455)
I20260811 11:20:04.223496 18406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.2:40357
W20260811 11:20:04.311645 18759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:04.311952 18759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:04.311983 18759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:04.313272 18759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:04.313324 18759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:04.313352 18759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.25.3
I20260811 11:20:04.314792 18759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.25.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.25.3
--webserver_port=0
--tserver_master_addrs=127.28.25.62:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.25.3
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:04.315107 18759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:04.315522 18759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:04.321575 18768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:04.321594 18759 server_base.cc:1034] running on GCE node
W20260811 11:20:04.321556 18766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:04.321555 18765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:04.322093 18759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:04.322450 18759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:04.323658 18759 hybrid_clock.cc:648] HybridClock initialized: now 1786447204323655 us; error 42 us; skew 500 ppm
I20260811 11:20:04.325376 18759 webserver.cc:460] Webserver started at http://127.28.25.3:43867/ using document root <none> and password file <none>
I20260811 11:20:04.325682 18759 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:04.325769 18759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:04.325932 18759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:04.327211 18759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/data/instance:
uuid: "8b5d0316e18b47ab9eb2d769a62d76fb"
format_stamp: "Formatted at 2026-08-11 11:20:04 on dist-test-slave-2w3w"
server_key: "b88e2cfc4c9ae6d887ad1029cd816bea"
server_key_iv: "4fc28e14bd03aaaaeb2ef57682231b2a"
server_key_version: "encryptionkey@0"
I20260811 11:20:04.327613 18759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance:
uuid: "8b5d0316e18b47ab9eb2d769a62d76fb"
format_stamp: "Formatted at 2026-08-11 11:20:04 on dist-test-slave-2w3w"
server_key: "b88e2cfc4c9ae6d887ad1029cd816bea"
server_key_iv: "4fc28e14bd03aaaaeb2ef57682231b2a"
server_key_version: "encryptionkey@0"
I20260811 11:20:04.329135 18759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:20:04.329989 18774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:04.330195 18759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:04.330330 18759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/wal
uuid: "8b5d0316e18b47ab9eb2d769a62d76fb"
format_stamp: "Formatted at 2026-08-11 11:20:04 on dist-test-slave-2w3w"
server_key: "b88e2cfc4c9ae6d887ad1029cd816bea"
server_key_iv: "4fc28e14bd03aaaaeb2ef57682231b2a"
server_key_version: "encryptionkey@0"
I20260811 11:20:04.330417 18759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:04.348677 18759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:04.349073 18759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:04.349277 18759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:04.349548 18759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:20:04.349946 18759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:20:04.350006 18759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:04.350051 18759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:20:04.350083 18759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:04.356899 18759 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.3:45175
I20260811 11:20:04.356999 18897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.3:45175 every 8 connection(s)
I20260811 11:20:04.357358 18759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/data/info.pb
I20260811 11:20:04.363570 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.358298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.3:38241 (local address 127.28.25.62:33109)
0811 11:20:04.358421 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:04.358424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:04.358737 (+ 313us) server_negotiation.cc:407] Connection header received
0811 11:20:04.359399 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:04.359400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:04.359413 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:04.359472 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:04.360325 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.360841 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.361913 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.362119 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.362211 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.363234 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:04.363236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:04.363236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:04.363243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:04.363258 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:04.363263 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:04.363326 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:04.363436 (+ 110us) server_negotiation.cc:299] Negotiation successful
0811 11:20:04.363480 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 11:20:04.363795 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.358252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:33109 (local address 127.28.25.3:38241)
0811 11:20:04.358574 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:04.358624 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:04.359285 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:04.359547 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:04.359552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:04.359563 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:04.360178 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.360185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.360964 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.360965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.361813 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.361816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.362235 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.362236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.362382 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.363066 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:20:04.363067 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:04.363099 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:20:04.363124 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:04.363340 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:04.363347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:04.363351 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:20:04.363437 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 11:20:04.363554 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":121,"threads_started":1}
I20260811 11:20:04.364420 18898 heartbeater.cc:347] Connected to a master server at 127.28.25.62:33109
I20260811 11:20:04.364528 18898 heartbeater.cc:464] Registering TS with master...
I20260811 11:20:04.364691 18898 heartbeater.cc:511] Master 127.28.25.62:33109 requested a full tablet report, sending...
I20260811 11:20:04.365150 18406 ts_manager.cc:194] Registered new tserver with Master: 8b5d0316e18b47ab9eb2d769a62d76fb (127.28.25.3:45175)
I20260811 11:20:04.365552 18406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.25.3:38241
I20260811 11:20:04.367355 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 18759
I20260811 11:20:04.367528 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/wal/instance
I20260811 11:20:04.367758 28772 external_mini_cluster.cc:1428] Setting key 92a406d666b0ccf2ad873a03e7ab41c0
I20260811 11:20:04.368999 28772 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:20:04.373571 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.370522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.28.25.62:33109)
0811 11:20:04.370605 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:04.370607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:04.370725 (+ 118us) server_negotiation.cc:407] Connection header received
0811 11:20:04.370753 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:04.370754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:04.370766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:04.370834 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:04.371061 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.371606 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.372375 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.372561 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.372619 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.373311 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:04.373312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:04.373312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:04.373317 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:04.373331 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:04.373335 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:04.373386 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:04.373487 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 11:20:04.373509 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 11:20:04.375320 18406 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:20:04.377095 18406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:20:04.384085 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.379779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.28.25.2:42455)
0811 11:20:04.380032 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:04.380037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:04.380170 (+ 133us) server_negotiation.cc:407] Connection header received
0811 11:20:04.380368 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:04.380370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:04.380388 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:04.380479 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:04.380645 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.381885 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.383026 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.383646 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.383733 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.383827 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:04.383944 (+ 117us) server_negotiation.cc:299] Negotiation successful
0811 11:20:04.383987 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":150,"threads_started":1}
I20260811 11:20:04.384104 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.379552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.2:42455 (local address 127.0.0.1:48946)
0811 11:20:04.380050 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:04.380085 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:04.380188 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:04.380485 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:04.380488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:04.380491 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:04.380568 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.380570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.381988 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.381989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.382905 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.382909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.383746 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.383747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.383829 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.383844 (+ 15us) client_negotiation.cc:769] Sending connection context
0811 11:20:04.383919 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 11:20:04.383981 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":91,"threads_started":1}
I20260811 11:20:04.384305 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.379782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.28.25.3:45175)
0811 11:20:04.380133 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:04.380137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:04.380509 (+ 372us) server_negotiation.cc:407] Connection header received
0811 11:20:04.380582 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:04.380585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:04.380603 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:04.380683 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:04.380790 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.381533 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.382916 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.383653 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.383726 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.383826 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:04.384157 (+ 331us) server_negotiation.cc:299] Negotiation successful
0811 11:20:04.384220 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":147,"threads_started":1}
I20260811 11:20:04.384565 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.379552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:45175 (local address 127.0.0.1:33142)
0811 11:20:04.380430 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:04.380439 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:04.380506 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:04.380679 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:04.380680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:04.380683 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:04.380726 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.380728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.381634 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.381635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.382811 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.382813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.383865 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.383866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.383928 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.383932 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:20:04.383988 (+ 56us) client_negotiation.cc:240] Negotiation successful
0811 11:20:04.384030 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":20,"thread_start_us":82,"threads_started":1}
I20260811 11:20:04.384769 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.379809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.28.25.1:44999)
0811 11:20:04.380222 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:04.380226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:04.380581 (+ 355us) server_negotiation.cc:407] Connection header received
0811 11:20:04.380631 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:04.380633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:04.380648 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:04.380714 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:04.380927 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.382020 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.383336 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.384150 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.384218 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.384321 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:04.384610 (+ 289us) server_negotiation.cc:299] Negotiation successful
0811 11:20:04.384666 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":144,"threads_started":1}
I20260811 11:20:04.384922 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.379560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:44999 (local address 127.0.0.1:50490)
0811 11:20:04.380302 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:04.380311 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:04.380355 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:04.380785 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:04.380786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:04.380789 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:04.380836 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.380838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.382155 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.382156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.383241 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.383243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.384362 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.384363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.384436 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.384441 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:20:04.384498 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 11:20:04.384543 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":104,"threads_started":1}
I20260811 11:20:04.385124 18683 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfd79bf99b14c2f9f6727ce34945033 (DEFAULT_TABLE table=test-table [id=483c318ea33942fdbdc0b899ca42c49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:04.385404 18828 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfd79bf99b14c2f9f6727ce34945033 (DEFAULT_TABLE table=test-table [id=483c318ea33942fdbdc0b899ca42c49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:04.385555 18683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfd79bf99b14c2f9f6727ce34945033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:04.385756 18828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfd79bf99b14c2f9f6727ce34945033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:04.385633 18541 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfd79bf99b14c2f9f6727ce34945033 (DEFAULT_TABLE table=test-table [id=483c318ea33942fdbdc0b899ca42c49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:20:04.385974 18541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfd79bf99b14c2f9f6727ce34945033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:04.388897 18914 tablet_bootstrap.cc:492] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: Bootstrap starting.
I20260811 11:20:04.389424 18915 tablet_bootstrap.cc:492] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: Bootstrap starting.
I20260811 11:20:04.389789 18914 tablet_bootstrap.cc:654] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:04.389967 18916 tablet_bootstrap.cc:492] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786: Bootstrap starting.
I20260811 11:20:04.390121 18915 tablet_bootstrap.cc:654] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:04.390151 18914 log.cc:824] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:04.390479 18915 log.cc:824] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:04.390703 18916 tablet_bootstrap.cc:654] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:04.391105 18914 tablet_bootstrap.cc:492] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: No bootstrap required, opened a new log
I20260811 11:20:04.391125 18916 log.cc:824] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:04.391196 18914 ts_tablet_manager.cc:1397] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:20:04.391551 18915 tablet_bootstrap.cc:492] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: No bootstrap required, opened a new log
I20260811 11:20:04.391650 18915 ts_tablet_manager.cc:1397] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:20:04.391937 18916 tablet_bootstrap.cc:492] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786: No bootstrap required, opened a new log
I20260811 11:20:04.392007 18916 ts_tablet_manager.cc:1397] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:20:04.394099 18914 raft_consensus.cc:348] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46013d595ff14c268435a975169273d3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42455 } } peers { permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 } } peers { permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 } }
I20260811 11:20:04.394273 18914 raft_consensus.cc:374] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:04.394363 18914 raft_consensus.cc:729] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5d0316e18b47ab9eb2d769a62d76fb, State: Initialized, Role: FOLLOWER
I20260811 11:20:04.394505 18914 consensus_queue.cc:260] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46013d595ff14c268435a975169273d3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42455 } } peers { permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 } } peers { permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 } }
I20260811 11:20:04.394685 18915 raft_consensus.cc:348] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46013d595ff14c268435a975169273d3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42455 } } peers { permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 } } peers { permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 } }
I20260811 11:20:04.394855 18914 ts_tablet_manager.cc:1428] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 11:20:04.394879 18915 raft_consensus.cc:374] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:04.394920 18915 raft_consensus.cc:729] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46013d595ff14c268435a975169273d3, State: Initialized, Role: FOLLOWER
I20260811 11:20:04.395015 18898 heartbeater.cc:503] Master 127.28.25.62:33109 was elected leader, sending a full tablet report...
I20260811 11:20:04.395076 18915 consensus_queue.cc:260] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46013d595ff14c268435a975169273d3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42455 } } peers { permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 } } peers { permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 } }
I20260811 11:20:04.395113 18916 raft_consensus.cc:348] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46013d595ff14c268435a975169273d3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42455 } } peers { permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 } } peers { permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 } }
I20260811 11:20:04.395264 18916 raft_consensus.cc:374] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:04.395306 18916 raft_consensus.cc:729] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3046bafff234ecd931af83b28c31786, State: Initialized, Role: FOLLOWER
I20260811 11:20:04.395335 18915 ts_tablet_manager.cc:1428] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260811 11:20:04.395481 18755 heartbeater.cc:503] Master 127.28.25.62:33109 was elected leader, sending a full tablet report...
I20260811 11:20:04.395457 18916 consensus_queue.cc:260] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46013d595ff14c268435a975169273d3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42455 } } peers { permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 } } peers { permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 } }
I20260811 11:20:04.395790 18612 heartbeater.cc:503] Master 127.28.25.62:33109 was elected leader, sending a full tablet report...
I20260811 11:20:04.396065 18916 ts_tablet_manager.cc:1428] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260811 11:20:04.465996 18757 tablet.cc:2369] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:20:04.579175 18613 tablet.cc:2369] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:20:04.608031 18899 tablet.cc:2369] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:20:04.616592 18921 raft_consensus.cc:482] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:20:04.616729 18921 raft_consensus.cc:504] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46013d595ff14c268435a975169273d3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42455 } } peers { permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 } } peers { permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 } }
I20260811 11:20:04.617071 18921 leader_election.cc:288] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3046bafff234ecd931af83b28c31786 (127.28.25.1:44999), 8b5d0316e18b47ab9eb2d769a62d76fb (127.28.25.3:45175)
I20260811 11:20:04.620306 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.617502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:57817 (local address 127.28.25.3:45175)
0811 11:20:04.617584 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:04.617586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:04.617594 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:20:04.617631 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:04.617632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:04.617640 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:04.617711 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:04.617916 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.618416 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.619517 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.619914 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.620008 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.620019 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:04.620179 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 11:20:04.620221 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:20:04.620306 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.617328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.3:45175 (local address 127.28.25.2:57817)
0811 11:20:04.617404 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:04.617421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:04.617500 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:04.617717 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:04.617718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:04.617720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:04.617801 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.617803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.618512 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.618513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.619416 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.619419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.620011 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.620012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.620097 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.620101 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:20:04.620172 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 11:20:04.620219 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 11:20:04.620661 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.617340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:44999 (local address 127.28.25.2:49831)
0811 11:20:04.617698 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:04.617711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:04.617814 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:04.617990 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:04.617991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:04.617993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:20:04.618077 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.618085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.618789 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.618791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.619714 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:04.619717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.620349 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.620351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:04.620444 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.620449 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:20:04.620522 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 11:20:04.620572 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":160,"thread_start_us":139,"threads_started":1}
I20260811 11:20:04.620671 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.617423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.25.2:49831 (local address 127.28.25.1:44999)
0811 11:20:04.617483 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:04.617485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:04.617827 (+ 342us) server_negotiation.cc:407] Connection header received
0811 11:20:04.617889 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:04.617890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:04.617904 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:04.617988 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:20:04.618167 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.618685 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.619822 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.620226 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.620333 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.620346 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:04.620553 (+ 207us) server_negotiation.cc:299] Negotiation successful
0811 11:20:04.620602 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260811 11:20:04.620712 18850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfd79bf99b14c2f9f6727ce34945033" candidate_uuid: "46013d595ff14c268435a975169273d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" is_pre_election: true
I20260811 11:20:04.620910 18850 raft_consensus.cc:2454] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46013d595ff14c268435a975169273d3 in term 0.
I20260811 11:20:04.621033 18562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfd79bf99b14c2f9f6727ce34945033" candidate_uuid: "46013d595ff14c268435a975169273d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3046bafff234ecd931af83b28c31786" is_pre_election: true
I20260811 11:20:04.621202 18562 raft_consensus.cc:2454] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46013d595ff14c268435a975169273d3 in term 0.
I20260811 11:20:04.621204 18631 leader_election.cc:302] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46013d595ff14c268435a975169273d3, 8b5d0316e18b47ab9eb2d769a62d76fb; no voters:
I20260811 11:20:04.621340 18921 raft_consensus.cc:2781] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:20:04.621408 18921 raft_consensus.cc:482] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:20:04.621436 18921 raft_consensus.cc:3037] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:04.622280 18921 raft_consensus.cc:504] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46013d595ff14c268435a975169273d3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42455 } } peers { permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 } } peers { permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 } }
I20260811 11:20:04.622465 18921 leader_election.cc:288] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [CANDIDATE]: Term 1 election: Requested vote from peers f3046bafff234ecd931af83b28c31786 (127.28.25.1:44999), 8b5d0316e18b47ab9eb2d769a62d76fb (127.28.25.3:45175)
I20260811 11:20:04.622601 18562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfd79bf99b14c2f9f6727ce34945033" candidate_uuid: "46013d595ff14c268435a975169273d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3046bafff234ecd931af83b28c31786"
I20260811 11:20:04.622625 18850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfd79bf99b14c2f9f6727ce34945033" candidate_uuid: "46013d595ff14c268435a975169273d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb"
I20260811 11:20:04.622730 18562 raft_consensus.cc:3037] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:04.622723 18850 raft_consensus.cc:3037] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:04.623633 18850 raft_consensus.cc:2454] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46013d595ff14c268435a975169273d3 in term 1.
I20260811 11:20:04.623638 18562 raft_consensus.cc:2454] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46013d595ff14c268435a975169273d3 in term 1.
I20260811 11:20:04.623831 18631 leader_election.cc:302] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46013d595ff14c268435a975169273d3, 8b5d0316e18b47ab9eb2d769a62d76fb; no voters:
I20260811 11:20:04.623937 18921 raft_consensus.cc:2781] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:04.624097 18921 raft_consensus.cc:686] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 1 LEADER]: Becoming Leader. State: Replica: 46013d595ff14c268435a975169273d3, State: Running, Role: LEADER
I20260811 11:20:04.624222 18921 consensus_queue.cc:237] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46013d595ff14c268435a975169273d3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42455 } } peers { permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 } } peers { permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 } }
I20260811 11:20:04.625212 18406 catalog_manager.cc:5495] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46013d595ff14c268435a975169273d3 (127.28.25.2). New cstate: current_term: 1 leader_uuid: "46013d595ff14c268435a975169273d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46013d595ff14c268435a975169273d3" member_type: VOTER last_known_addr { host: "127.28.25.2" port: 42455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 } health_report { overall_health: UNKNOWN } } }
I20260811 11:20:04.670286 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.667388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.28.25.2:42455)
0811 11:20:04.667439 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:04.667441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:04.667459 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:20:04.667545 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:04.667546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:04.667567 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:04.667627 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:04.667737 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.668201 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.669272 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:04.669496 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:04.669562 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:04.669877 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:04.670006 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:04.670092 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:04.670159 (+ 67us) server_negotiation.cc:299] Negotiation successful
0811 11:20:04.670183 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 11:20:04.672353 18562 raft_consensus.cc:1261] T 9cfd79bf99b14c2f9f6727ce34945033 P f3046bafff234ecd931af83b28c31786 [term 1 FOLLOWER]: Refusing update from remote peer 46013d595ff14c268435a975169273d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:04.672371 18850 raft_consensus.cc:1261] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb [term 1 FOLLOWER]: Refusing update from remote peer 46013d595ff14c268435a975169273d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:04.672806 18921 consensus_queue.cc:1035] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3046bafff234ecd931af83b28c31786" member_type: VOTER last_known_addr { host: "127.28.25.1" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:04.672995 18924 consensus_queue.cc:1035] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b5d0316e18b47ab9eb2d769a62d76fb" member_type: VOTER last_known_addr { host: "127.28.25.3" port: 45175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:04.675586 18928 mvcc.cc:205] Tried to move back new op lower bound from 7317287750334955520 to 7317287750141632512. Current Snapshot: MvccSnapshot[applied={T|T < 7317287750334955520}]
I20260811 11:20:04.675796 18929 mvcc.cc:205] Tried to move back new op lower bound from 7317287750334955520 to 7317287750141632512. Current Snapshot: MvccSnapshot[applied={T|T < 7317287750334955520}]
I20260811 11:20:04.675853 18931 mvcc.cc:205] Tried to move back new op lower bound from 7317287750334955520 to 7317287750141632512. Current Snapshot: MvccSnapshot[applied={T|T < 7317287750334955520}]
I20260811 11:20:04.678920 18406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:20:04.679075 18406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:20:04.679982 18406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d2784714a3642d4a9d967b1f042c6e5: Sending DeleteTablet for 3 replicas of tablet 9cfd79bf99b14c2f9f6727ce34945033
I20260811 11:20:04.680351 18683 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfd79bf99b14c2f9f6727ce34945033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:04 UTC) from {username='slave'} at 127.0.0.1:48946
I20260811 11:20:04.680410 18828 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfd79bf99b14c2f9f6727ce34945033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:04 UTC) from {username='slave'} at 127.0.0.1:33142
I20260811 11:20:04.680661 18939 tablet_replica.cc:330] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: stopping tablet replica
I20260811 11:20:04.680811 18939 raft_consensus.cc:2229] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:20:04.680787 18541 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfd79bf99b14c2f9f6727ce34945033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:20:04 UTC) from {username='slave'} at 127.0.0.1:50490
I20260811 11:20:04.680840 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 18474
I20260811 11:20:04.680994 18939 raft_consensus.cc:2258] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:20:04.688856 18940 tablet_replica.cc:330] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: stopping tablet replica
I20260811 11:20:04.688900 18939 ts_tablet_manager.cc:1905] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:20:04.689108 18940 raft_consensus.cc:2229] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:20:04.689304 18940 raft_consensus.cc:2258] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:20:04.689993 18940 ts_tablet_manager.cc:1905] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:20:04.690745 18939 ts_tablet_manager.cc:1918] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:20:04.690840 18939 log.cc:1196] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-1/wal/wals/9cfd79bf99b14c2f9f6727ce34945033
I20260811 11:20:04.691171 18939 ts_tablet_manager.cc:1939] T 9cfd79bf99b14c2f9f6727ce34945033 P 46013d595ff14c268435a975169273d3: Deleting consensus metadata
I20260811 11:20:04.691733 18389 catalog_manager.cc:4841] TS 46013d595ff14c268435a975169273d3 (127.28.25.2:42455): tablet 9cfd79bf99b14c2f9f6727ce34945033 (table test-table [id=483c318ea33942fdbdc0b899ca42c49b]) successfully deleted
I20260811 11:20:04.692355 18940 ts_tablet_manager.cc:1918] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:20:04.692448 18940 log.cc:1196] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: Deleting WAL directory at /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786446911996770-28772-0/minicluster-data/ts-2/wal/wals/9cfd79bf99b14c2f9f6727ce34945033
I20260811 11:20:04.692770 18940 ts_tablet_manager.cc:1939] T 9cfd79bf99b14c2f9f6727ce34945033 P 8b5d0316e18b47ab9eb2d769a62d76fb: Deleting consensus metadata
I20260811 11:20:04.693282 18388 catalog_manager.cc:4841] TS 8b5d0316e18b47ab9eb2d769a62d76fb (127.28.25.3:45175): tablet 9cfd79bf99b14c2f9f6727ce34945033 (table test-table [id=483c318ea33942fdbdc0b899ca42c49b]) successfully deleted
W20260811 11:20:04.694262 18391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.25.1:44999 (error 108)
I20260811 11:20:04.694471 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 18615
I20260811 11:20:04.698467 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:04.698246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.1:44999 (local address 127.0.0.1:50494)
0811 11:20:04.698308 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:04.698386 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.1:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260811 11:20:04.698647 18391 catalog_manager.cc:4584] TS f3046bafff234ecd931af83b28c31786 (127.28.25.1:44999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cfd79bf99b14c2f9f6727ce34945033: Network error: Client connection negotiation failed: client connection to 127.28.25.1:44999: connect: Connection refused (error 111)
I20260811 11:20:04.702894 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 18759
I20260811 11:20:04.710424 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 18371
2026-08-11T11:20:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (996 ms)
[----------] 6 tests from EncryptionPolicyTest (9877 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:04 dist-test-slave-2w3w krb5kdc[18945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:04 dist-test-slave-2w3w krb5kdc[18945](info): set up 2 sockets
Aug 11 11:20:04 dist-test-slave-2w3w krb5kdc[18945](info): commencing operation
krb5kdc: starting...
W20260811 11:20:05.818361 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:05 dist-test-slave-2w3w krb5kdc[18945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:05.959357 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:06.052798 18959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:06.053092 18959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:06.053138 18959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:06.054487 18959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:06.054580 18959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:06.054610 18959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:06.054642 18959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:06.054672 18959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:06.056908 18959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:06.057286 18959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:06.057601 18959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:06.064327 18966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:06.064429 18969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:06.064512 18959 server_base.cc:1034] running on GCE node
W20260811 11:20:06.064321 18965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:06.065147 18959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:06.065639 18959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:06.066764 18959 hybrid_clock.cc:648] HybridClock initialized: now 1786447206066733 us; error 49 us; skew 500 ppm
Aug 11 11:20:06 dist-test-slave-2w3w krb5kdc[18945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:06.070677 18959 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:06.071346 18959 webserver.cc:460] Webserver started at http://127.0.0.1:44157/ using document root <none> and password file <none>
I20260811 11:20:06.071637 18959 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:06.071712 18959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:06.071895 18959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:06.073616 18959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "52fcbdec7b874c85a70591889a8d8dc5"
format_stamp: "Formatted at 2026-08-11 11:20:06 on dist-test-slave-2w3w"
server_key: "92e9a262a6a0b1a0355c845160ad7616"
server_key_iv: "c0f62d1275fa9d9af756c314c960208c"
server_key_version: "encryptionkey@0"
I20260811 11:20:06.074044 18959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "52fcbdec7b874c85a70591889a8d8dc5"
format_stamp: "Formatted at 2026-08-11 11:20:06 on dist-test-slave-2w3w"
server_key: "92e9a262a6a0b1a0355c845160ad7616"
server_key_iv: "c0f62d1275fa9d9af756c314c960208c"
server_key_version: "encryptionkey@0"
I20260811 11:20:06.075753 18959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260811 11:20:06.076905 18980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:06.077095 18959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:06.077204 18959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "52fcbdec7b874c85a70591889a8d8dc5"
format_stamp: "Formatted at 2026-08-11 11:20:06 on dist-test-slave-2w3w"
server_key: "92e9a262a6a0b1a0355c845160ad7616"
server_key_iv: "c0f62d1275fa9d9af756c314c960208c"
server_key_version: "encryptionkey@0"
I20260811 11:20:06.077312 18959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:06.099776 18959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:06.102417 18959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:06.102638 18959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:06.107018 18959 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33797
I20260811 11:20:06.107070 19036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33797 every 8 connection(s)
I20260811 11:20:06.107481 18959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:06.108489 19037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:06.111104 19037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5: Bootstrap starting.
I20260811 11:20:06.111765 19037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:06.112084 19037 log.cc:824] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:06.113037 19037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5: No bootstrap required, opened a new log
I20260811 11:20:06.114786 19037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fcbdec7b874c85a70591889a8d8dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33797 } }
I20260811 11:20:06.114943 19037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:06.115046 19037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fcbdec7b874c85a70591889a8d8dc5, State: Initialized, Role: FOLLOWER
I20260811 11:20:06.115236 19037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fcbdec7b874c85a70591889a8d8dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33797 } }
I20260811 11:20:06.115307 19037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:06.115331 19037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:06.115401 19037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:06.116045 19037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fcbdec7b874c85a70591889a8d8dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33797 } }
I20260811 11:20:06.116139 19037 leader_election.cc:302] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52fcbdec7b874c85a70591889a8d8dc5; no voters:
I20260811 11:20:06.116377 19037 leader_election.cc:288] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:06.116472 19041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:06.116482 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 18959
I20260811 11:20:06.116613 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:06.116648 19041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 52fcbdec7b874c85a70591889a8d8dc5, State: Running, Role: LEADER
I20260811 11:20:06.116707 19037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:06.116824 19041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fcbdec7b874c85a70591889a8d8dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33797 } }
I20260811 11:20:06.116868 28772 external_mini_cluster.cc:1428] Setting key b8c388488c8a9b8a1f76ae7b4a875c3c
I20260811 11:20:06.117347 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52fcbdec7b874c85a70591889a8d8dc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fcbdec7b874c85a70591889a8d8dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33797 } } }
I20260811 11:20:06.117432 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:06.117702 19043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52fcbdec7b874c85a70591889a8d8dc5. Latest consensus state: current_term: 1 leader_uuid: "52fcbdec7b874c85a70591889a8d8dc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fcbdec7b874c85a70591889a8d8dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33797 } } }
I20260811 11:20:06.117842 19043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:06.117920 19053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:06.118489 19053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:06.120234 19053 catalog_manager.cc:1347] Generated new cluster ID: f952f112b22544e7b617d161fca5cb86
I20260811 11:20:06.120288 19053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 11:20:06 dist-test-slave-2w3w krb5kdc[18945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 11:20:06.128351 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:06.117432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48943 (local address 127.0.0.1:33797)
0811 11:20:06.117652 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:06.117663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:06.117703 (+ 40us) server_negotiation.cc:407] Connection header received
0811 11:20:06.117850 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:06.117858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:06.117926 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:06.119024 (+ 1098us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:06.119129 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:06.120539 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:06.121385 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:06.121836 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:06.121964 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:06.124648 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:06.124656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:06.124656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:06.124674 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:06.126554 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:06.127029 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:06.127040 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:06.127040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:06.127155 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:06.127405 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:06.127406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:06.127406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:06.127688 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:06.127903 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:06.127996 (+ 93us) server_negotiation.cc:299] Negotiation successful
0811 11:20:06.128109 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260811 11:20:06.129585 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:06.137936 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:06.131137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54957 (local address 127.0.0.1:33797)
0811 11:20:06.131180 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:06.131183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:06.131320 (+ 137us) server_negotiation.cc:407] Connection header received
0811 11:20:06.131397 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:06.131398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:06.131408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:06.131464 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:06.132032 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:06.132567 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:06.133345 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:06.133587 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:06.133679 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:06.135113 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:06.135115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:06.135115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:06.135122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:06.136647 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:06.137058 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:06.137059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:06.137060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:06.137123 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:06.137362 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:06.137363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:06.137363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:06.137487 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:06.137610 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:06.137741 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 11:20:06.137818 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 11:20:06.144017 19053 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:06.144757 19053 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:06.155622 19053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52fcbdec7b874c85a70591889a8d8dc5: Generated new TSK 0
I20260811 11:20:06.155937 19053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:06.163353 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 18959
W20260811 11:20:06.171911 18943 connection.cc:535] client connection to 127.0.0.1:33797 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33797): Connection reset by peer (error 104)
2026-08-11T11:20:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:06 dist-test-slave-2w3w krb5kdc[19071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:06 dist-test-slave-2w3w krb5kdc[19071](info): set up 2 sockets
Aug 11 11:20:06 dist-test-slave-2w3w krb5kdc[19071](info): commencing operation
krb5kdc: starting...
W20260811 11:20:07.286446 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:07 dist-test-slave-2w3w krb5kdc[19071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:07.431699 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:07.519776 19085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:07.520036 19085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:07.520064 19085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:07.521361 19085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:07.521422 19085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:07.521437 19085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:07.521451 19085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:07.521463 19085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:07.523222 19085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:07.523475 19085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:07.523690 19085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:07.529469 19093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:07.529446 19095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:07.529677 19085 server_base.cc:1034] running on GCE node
W20260811 11:20:07.529538 19092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:07.530085 19085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:07.530470 19085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:07.531672 19085 hybrid_clock.cc:648] HybridClock initialized: now 1786447207531631 us; error 80 us; skew 500 ppm
Aug 11 11:20:07 dist-test-slave-2w3w krb5kdc[19071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:07.534665 19085 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:07.535398 19085 webserver.cc:460] Webserver started at http://127.0.0.1:39525/ using document root <none> and password file <none>
I20260811 11:20:07.535663 19085 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:07.535738 19085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:07.535908 19085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:07.537120 19085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "d3affad69f73407c806575e2c100182b"
format_stamp: "Formatted at 2026-08-11 11:20:07 on dist-test-slave-2w3w"
server_key: "f24bf5ce5a8666178c276e502dc32810"
server_key_iv: "f1dd34632e9688fe90dcb5dbe1cd4954"
server_key_version: "encryptionkey@0"
I20260811 11:20:07.537541 19085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "d3affad69f73407c806575e2c100182b"
format_stamp: "Formatted at 2026-08-11 11:20:07 on dist-test-slave-2w3w"
server_key: "f24bf5ce5a8666178c276e502dc32810"
server_key_iv: "f1dd34632e9688fe90dcb5dbe1cd4954"
server_key_version: "encryptionkey@0"
I20260811 11:20:07.539209 19085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:20:07.540161 19103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:07.540409 19085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:20:07.540551 19085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "d3affad69f73407c806575e2c100182b"
format_stamp: "Formatted at 2026-08-11 11:20:07 on dist-test-slave-2w3w"
server_key: "f24bf5ce5a8666178c276e502dc32810"
server_key_iv: "f1dd34632e9688fe90dcb5dbe1cd4954"
server_key_version: "encryptionkey@0"
I20260811 11:20:07.540639 19085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:07.554242 19085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:07.557346 19085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:07.557538 19085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:07.563217 19085 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41087
I20260811 11:20:07.563266 19159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41087 every 8 connection(s)
I20260811 11:20:07.563735 19085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:07.564750 19160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:07.567559 19160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b: Bootstrap starting.
I20260811 11:20:07.568516 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 19085
I20260811 11:20:07.568555 19160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:07.568665 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:07.568917 28772 external_mini_cluster.cc:1428] Setting key d861dfe470ac4c3da60d447a07e9023a
I20260811 11:20:07.568950 19160 log.cc:824] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:07.570013 19160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b: No bootstrap required, opened a new log
I20260811 11:20:07.572378 19160 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3affad69f73407c806575e2c100182b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41087 } }
I20260811 11:20:07.572598 19160 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:07.572634 19160 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3affad69f73407c806575e2c100182b, State: Initialized, Role: FOLLOWER
I20260811 11:20:07.572777 19160 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3affad69f73407c806575e2c100182b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41087 } }
I20260811 11:20:07.572852 19160 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:07.572882 19160 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:07.572922 19160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:07.573654 19160 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3affad69f73407c806575e2c100182b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41087 } }
I20260811 11:20:07.573799 19160 leader_election.cc:302] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3affad69f73407c806575e2c100182b; no voters:
I20260811 11:20:07.574057 19160 leader_election.cc:288] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:07.574081 19166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:07.574379 19160 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:07.574436 19166 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [term 1 LEADER]: Becoming Leader. State: Replica: d3affad69f73407c806575e2c100182b, State: Running, Role: LEADER
I20260811 11:20:07.574522 19166 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3affad69f73407c806575e2c100182b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41087 } }
I20260811 11:20:07.575146 19167 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3affad69f73407c806575e2c100182b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3affad69f73407c806575e2c100182b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41087 } } }
I20260811 11:20:07.575488 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:07.569494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57893 (local address 127.0.0.1:41087)
0811 11:20:07.569678 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:07.569687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:07.569718 (+ 31us) server_negotiation.cc:407] Connection header received
0811 11:20:07.570314 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:07.570323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:07.570385 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:07.570511 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:07.570819 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:07.571918 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:07.573043 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:07.573497 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:07.573596 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:07.574452 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:07.574457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:07.574457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:07.574685 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:07.574731 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:07.574744 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:07.574856 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:07.575200 (+ 344us) server_negotiation.cc:299] Negotiation successful
0811 11:20:07.575232 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260811 11:20:07.575544 19168 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3affad69f73407c806575e2c100182b. Latest consensus state: current_term: 1 leader_uuid: "d3affad69f73407c806575e2c100182b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3affad69f73407c806575e2c100182b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41087 } } }
I20260811 11:20:07.575608 19168 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:07.575294 19167 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b [sys.catalog]: This master's current role is: LEADER
W20260811 11:20:07.576566 19184 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:07.576622 19184 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:07.576676 19181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:07.576843 19181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:07.576861 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:07.578858 19181 catalog_manager.cc:1347] Generated new cluster ID: 9a3377827ddf463bb151f5aa2f64bb9d
I20260811 11:20:07.578919 19181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:07.582552 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:07.578501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48899 (local address 127.0.0.1:41087)
0811 11:20:07.578566 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:07.578568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:07.578584 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:20:07.578656 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:07.578657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:07.578669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:07.578756 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:07.578948 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:07.579760 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:07.580909 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:07.581262 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:07.581317 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:07.582217 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:07.582219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:07.582220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:07.582226 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:07.582243 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:07.582248 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:07.582282 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:07.582448 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 11:20:07.582463 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:20:07.598918 19181 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:07.599543 19181 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:07.606352 19181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3affad69f73407c806575e2c100182b: Generated new TSK 0
I20260811 11:20:07.606581 19181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:07.616521 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 19085
W20260811 11:20:07.624714 19068 connection.cc:535] client connection to 127.0.0.1:41087 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41087): Connection reset by peer (error 104)
2026-08-11T11:20:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:07 dist-test-slave-2w3w krb5kdc[19194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:07 dist-test-slave-2w3w krb5kdc[19194](info): set up 2 sockets
Aug 11 11:20:07 dist-test-slave-2w3w krb5kdc[19194](info): commencing operation
krb5kdc: starting...
W20260811 11:20:08.745885 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:08 dist-test-slave-2w3w krb5kdc[19194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:08.886940 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:08.975220 19208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:08.975544 19208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:08.975592 19208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:08.976918 19208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:08.977001 19208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:08.977027 19208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:08.977068 19208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:08.977097 19208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:08.978747 19208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:08.979105 19208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:08.979379 19208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:08.985136 19213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:08.985146 19214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:08.985155 19217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:08.985392 19208 server_base.cc:1034] running on GCE node
I20260811 11:20:08.985664 19208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:08.985980 19208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:08.987154 19208 hybrid_clock.cc:648] HybridClock initialized: now 1786447208987130 us; error 32 us; skew 500 ppm
Aug 11 11:20:08 dist-test-slave-2w3w krb5kdc[19194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:08.990108 19208 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:08.990693 19208 webserver.cc:460] Webserver started at http://127.0.0.1:43603/ using document root <none> and password file <none>
I20260811 11:20:08.991056 19208 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:08.991115 19208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:08.991287 19208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:08.992561 19208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "62c7a88ad134469380a73c031db27c17"
format_stamp: "Formatted at 2026-08-11 11:20:08 on dist-test-slave-2w3w"
server_key: "e5ae74bff06ccc8db6b47c6fe1006c05"
server_key_iv: "6a6b68846546d224b4034f00d25f0cd0"
server_key_version: "encryptionkey@0"
I20260811 11:20:08.992975 19208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "62c7a88ad134469380a73c031db27c17"
format_stamp: "Formatted at 2026-08-11 11:20:08 on dist-test-slave-2w3w"
server_key: "e5ae74bff06ccc8db6b47c6fe1006c05"
server_key_iv: "6a6b68846546d224b4034f00d25f0cd0"
server_key_version: "encryptionkey@0"
I20260811 11:20:08.994570 19208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:20:08.995600 19226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:08.995817 19208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 11:20:08.995903 19208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "62c7a88ad134469380a73c031db27c17"
format_stamp: "Formatted at 2026-08-11 11:20:08 on dist-test-slave-2w3w"
server_key: "e5ae74bff06ccc8db6b47c6fe1006c05"
server_key_iv: "6a6b68846546d224b4034f00d25f0cd0"
server_key_version: "encryptionkey@0"
I20260811 11:20:08.995993 19208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:08.999850 19208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:09.002664 19208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:09.002926 19208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:09.007843 19208 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45039
I20260811 11:20:09.007942 19283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45039 every 8 connection(s)
I20260811 11:20:09.008314 19208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:09.009460 19284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:09.012200 19284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17: Bootstrap starting.
I20260811 11:20:09.012887 19284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:09.013223 19284 log.cc:824] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:09.013661 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 19208
I20260811 11:20:09.013768 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:09.014029 28772 external_mini_cluster.cc:1428] Setting key cf845e95da46e6a79c9e5645cb2a462f
I20260811 11:20:09.014285 19284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17: No bootstrap required, opened a new log
I20260811 11:20:09.016937 19284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c7a88ad134469380a73c031db27c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45039 } }
I20260811 11:20:09.017150 19284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:09.017220 19284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c7a88ad134469380a73c031db27c17, State: Initialized, Role: FOLLOWER
I20260811 11:20:09.017359 19284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c7a88ad134469380a73c031db27c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45039 } }
I20260811 11:20:09.017499 19284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:09.017541 19284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:09.017599 19284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:09.017717 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:09.014669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34951 (local address 127.0.0.1:45039)
0811 11:20:09.014909 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:09.014920 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:09.015035 (+ 115us) server_negotiation.cc:407] Connection header received
0811 11:20:09.015943 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:09.015955 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:09.015998 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:09.016179 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:09.016977 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:09.016982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:09.016982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:09.016987 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:09.017035 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:09.017050 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:09.017130 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:09.017246 (+ 116us) server_negotiation.cc:299] Negotiation successful
0811 11:20:09.017292 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":121,"threads_started":1}
I20260811 11:20:09.018522 19284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c7a88ad134469380a73c031db27c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45039 } }
I20260811 11:20:09.018642 19284 leader_election.cc:302] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62c7a88ad134469380a73c031db27c17; no voters:
I20260811 11:20:09.018890 19284 leader_election.cc:288] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:09.018997 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:09.019047 19289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:09.019208 19289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [term 1 LEADER]: Becoming Leader. State: Replica: 62c7a88ad134469380a73c031db27c17, State: Running, Role: LEADER
I20260811 11:20:09.019240 19284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:09.019300 19289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c7a88ad134469380a73c031db27c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45039 } }
I20260811 11:20:09.019677 19289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62c7a88ad134469380a73c031db27c17. Latest consensus state: current_term: 1 leader_uuid: "62c7a88ad134469380a73c031db27c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c7a88ad134469380a73c031db27c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45039 } } }
I20260811 11:20:09.019745 19289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:09.019793 19290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62c7a88ad134469380a73c031db27c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c7a88ad134469380a73c031db27c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45039 } } }
I20260811 11:20:09.019887 19290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17 [sys.catalog]: This master's current role is: LEADER
W20260811 11:20:09.021317 19309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:09.021384 19309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:09.021467 19296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:09.021668 19296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:09.023294 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:09.020830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47985 (local address 127.0.0.1:45039)
0811 11:20:09.020869 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:09.020871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:09.021250 (+ 379us) server_negotiation.cc:407] Connection header received
0811 11:20:09.021283 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:09.021284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:09.021291 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:09.021382 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:09.022922 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:09.022923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:09.022924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:09.022929 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:09.022952 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:09.023058 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:09.023125 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:09.023213 (+ 88us) server_negotiation.cc:299] Negotiation successful
0811 11:20:09.023229 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260811 11:20:09.023682 19296 catalog_manager.cc:1347] Generated new cluster ID: e2d5bca3ce40438bb5d8ac3fb9f71730
I20260811 11:20:09.023736 19296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:09.029337 19296 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:09.029569 19296 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:09.032359 19296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62c7a88ad134469380a73c031db27c17: Generated new TSK 0
I20260811 11:20:09.032581 19296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:09.037853 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 19208
2026-08-11T11:20:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:09 dist-test-slave-2w3w krb5kdc[19316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:09 dist-test-slave-2w3w krb5kdc[19316](info): set up 2 sockets
Aug 11 11:20:09 dist-test-slave-2w3w krb5kdc[19316](info): commencing operation
krb5kdc: starting...
W20260811 11:20:10.175976 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:10 dist-test-slave-2w3w krb5kdc[19316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:10.322696 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:10.409091 19330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:10.409382 19330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:10.409427 19330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:10.410701 19330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:10.410804 19330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:10.410827 19330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:10.410847 19330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:10.410874 19330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:10.412601 19330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:10.412947 19330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:10.413208 19330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:10.418762 19338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:10.418823 19339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:10.418805 19341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:10.418907 19330 server_base.cc:1034] running on GCE node
I20260811 11:20:10.419312 19330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:10.419689 19330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:10.420872 19330 hybrid_clock.cc:648] HybridClock initialized: now 1786447210420842 us; error 52 us; skew 500 ppm
Aug 11 11:20:10 dist-test-slave-2w3w krb5kdc[19316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:10.423960 19330 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:10.424540 19330 webserver.cc:460] Webserver started at http://127.0.0.1:35511/ using document root <none> and password file <none>
I20260811 11:20:10.424801 19330 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:10.424877 19330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:10.425050 19330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:10.426229 19330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "ca6b358b635549a1845ee3c5c3b35284"
format_stamp: "Formatted at 2026-08-11 11:20:10 on dist-test-slave-2w3w"
server_key: "d02e09c00aa32e51de979acba714f74a"
server_key_iv: "ab0d5ce64d4541bedfedf31275f9aeeb"
server_key_version: "encryptionkey@0"
I20260811 11:20:10.426627 19330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "ca6b358b635549a1845ee3c5c3b35284"
format_stamp: "Formatted at 2026-08-11 11:20:10 on dist-test-slave-2w3w"
server_key: "d02e09c00aa32e51de979acba714f74a"
server_key_iv: "ab0d5ce64d4541bedfedf31275f9aeeb"
server_key_version: "encryptionkey@0"
I20260811 11:20:10.428285 19330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:20:10.429348 19349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:10.429703 19330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:10.429867 19330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "ca6b358b635549a1845ee3c5c3b35284"
format_stamp: "Formatted at 2026-08-11 11:20:10 on dist-test-slave-2w3w"
server_key: "d02e09c00aa32e51de979acba714f74a"
server_key_iv: "ab0d5ce64d4541bedfedf31275f9aeeb"
server_key_version: "encryptionkey@0"
I20260811 11:20:10.429960 19330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:10.451134 19330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:10.453579 19330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:10.453783 19330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:10.458472 19330 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45937
I20260811 11:20:10.458511 19403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45937 every 8 connection(s)
I20260811 11:20:10.458910 19330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:10.459453 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 19330
I20260811 11:20:10.459597 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:10.459812 28772 external_mini_cluster.cc:1428] Setting key fa0423ea2089047bf4bdb0e18d3edd60
I20260811 11:20:10.460160 19404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:10.463403 19404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284: Bootstrap starting.
I20260811 11:20:10.464066 19404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:10.464402 19404 log.cc:824] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:10.465309 19404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284: No bootstrap required, opened a new log
Aug 11 11:20:10 dist-test-slave-2w3w krb5kdc[19316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 11:20:10.467850 19404 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6b358b635549a1845ee3c5c3b35284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260811 11:20:10.468042 19404 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:10.468068 19404 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6b358b635549a1845ee3c5c3b35284, State: Initialized, Role: FOLLOWER
I20260811 11:20:10.468218 19404 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6b358b635549a1845ee3c5c3b35284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260811 11:20:10.468317 19404 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:10.468356 19404 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:10.468407 19404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:10.469126 19404 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6b358b635549a1845ee3c5c3b35284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260811 11:20:10.469254 19404 leader_election.cc:302] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca6b358b635549a1845ee3c5c3b35284; no voters:
I20260811 11:20:10.469498 19404 leader_election.cc:288] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:10.469574 19409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:10.469753 19409 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [term 1 LEADER]: Becoming Leader. State: Replica: ca6b358b635549a1845ee3c5c3b35284, State: Running, Role: LEADER
I20260811 11:20:10.469810 19404 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:10.469925 19409 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6b358b635549a1845ee3c5c3b35284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260811 11:20:10.470355 19409 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6b358b635549a1845ee3c5c3b35284. Latest consensus state: current_term: 1 leader_uuid: "ca6b358b635549a1845ee3c5c3b35284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6b358b635549a1845ee3c5c3b35284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } } }
I20260811 11:20:10.470445 19409 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:10.471123 19416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:10.471632 19410 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca6b358b635549a1845ee3c5c3b35284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6b358b635549a1845ee3c5c3b35284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } } }
I20260811 11:20:10.471721 19410 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:10.471918 19416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:10.473024 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:10.460373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57723 (local address 127.0.0.1:45937)
0811 11:20:10.460587 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:10.460604 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:10.460632 (+ 28us) server_negotiation.cc:407] Connection header received
0811 11:20:10.461331 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:10.461339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:10.461430 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:10.461592 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:10.462287 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:10.463277 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:10.464134 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:10.464677 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:10.464801 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:10.467932 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:10.467936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:10.467936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:10.467948 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:10.470506 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:10.471078 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:10.471081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:10.471082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:10.471202 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:10.471482 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:10.471483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:10.471483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:10.471857 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:10.472574 (+ 717us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:10.472660 (+ 86us) server_negotiation.cc:299] Negotiation successful
0811 11:20:10.472841 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260811 11:20:10.474009 19416 catalog_manager.cc:1347] Generated new cluster ID: a2a7211a09684176b23600368e84b291
I20260811 11:20:10.474076 19416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:10.474190 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:10.487710 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:10.478631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39339 (local address 127.0.0.1:45937)
0811 11:20:10.478934 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:10.478936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:10.479174 (+ 238us) server_negotiation.cc:407] Connection header received
0811 11:20:10.479357 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:10.479359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:10.479376 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:10.479467 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:10.480268 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:10.481077 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:10.482224 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:10.482586 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:10.482668 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:10.484460 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:10.484462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:10.484462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:10.484472 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:10.486321 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:10.486780 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:10.486781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:10.486782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:10.486853 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:10.487098 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:10.487098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:10.487099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:10.487254 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:10.487361 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:10.487510 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 11:20:10.487603 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260811 11:20:10.494794 19416 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:10.495478 19416 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:10.509452 19416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca6b358b635549a1845ee3c5c3b35284: Generated new TSK 0
I20260811 11:20:10.509718 19416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:10.517957 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 19330
2026-08-11T11:20:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:10 dist-test-slave-2w3w krb5kdc[19437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:10 dist-test-slave-2w3w krb5kdc[19437](info): set up 2 sockets
Aug 11 11:20:10 dist-test-slave-2w3w krb5kdc[19437](info): commencing operation
krb5kdc: starting...
W20260811 11:20:11.635499 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:11 dist-test-slave-2w3w krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:11.780061 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:11.866843 19451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:11.867152 19451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:11.867199 19451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:11.868512 19451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:11.868599 19451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:11.868624 19451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:11.868645 19451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:11.868677 19451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:11.870453 19451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:11.870807 19451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:11.871100 19451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:11.876444 19459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:11.876444 19456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:11.876477 19451 server_base.cc:1034] running on GCE node
W20260811 11:20:11.876456 19457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:11.877007 19451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:11.877373 19451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:11.878535 19451 hybrid_clock.cc:648] HybridClock initialized: now 1786447211878508 us; error 44 us; skew 500 ppm
Aug 11 11:20:11 dist-test-slave-2w3w krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:11.885339 19451 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:11.885874 19451 webserver.cc:460] Webserver started at http://127.0.0.1:45365/ using document root <none> and password file <none>
I20260811 11:20:11.886147 19451 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:11.886245 19451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:11.886397 19451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:11.887603 19451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "f1b1c2a549b041f6b108b94ff3f32beb"
format_stamp: "Formatted at 2026-08-11 11:20:11 on dist-test-slave-2w3w"
server_key: "def30b7633eccb7ba8eaba64392d9278"
server_key_iv: "b16ab1de85e21daa23bfa5211bd58ba5"
server_key_version: "encryptionkey@0"
I20260811 11:20:11.887982 19451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "f1b1c2a549b041f6b108b94ff3f32beb"
format_stamp: "Formatted at 2026-08-11 11:20:11 on dist-test-slave-2w3w"
server_key: "def30b7633eccb7ba8eaba64392d9278"
server_key_iv: "b16ab1de85e21daa23bfa5211bd58ba5"
server_key_version: "encryptionkey@0"
I20260811 11:20:11.889432 19451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:20:11.890292 19467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:11.890563 19451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:11.890695 19451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "f1b1c2a549b041f6b108b94ff3f32beb"
format_stamp: "Formatted at 2026-08-11 11:20:11 on dist-test-slave-2w3w"
server_key: "def30b7633eccb7ba8eaba64392d9278"
server_key_iv: "b16ab1de85e21daa23bfa5211bd58ba5"
server_key_version: "encryptionkey@0"
I20260811 11:20:11.890786 19451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:11.899672 19451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:11.901927 19451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:11.902109 19451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:11.906741 19451 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41351
I20260811 11:20:11.906817 19522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41351 every 8 connection(s)
I20260811 11:20:11.907284 19451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:11.908264 19524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:11.910895 19524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb: Bootstrap starting.
I20260811 11:20:11.911675 19524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:11.912062 19524 log.cc:824] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:11.913013 19524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb: No bootstrap required, opened a new log
I20260811 11:20:11.914916 19524 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b1c2a549b041f6b108b94ff3f32beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260811 11:20:11.915158 19524 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:11.915189 19524 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b1c2a549b041f6b108b94ff3f32beb, State: Initialized, Role: FOLLOWER
I20260811 11:20:11.915360 19524 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b1c2a549b041f6b108b94ff3f32beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260811 11:20:11.915431 19524 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:11.915475 19524 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:11.915534 19524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:11.916242 19524 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b1c2a549b041f6b108b94ff3f32beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260811 11:20:11.916343 19524 leader_election.cc:302] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1b1c2a549b041f6b108b94ff3f32beb; no voters:
I20260811 11:20:11.916558 19524 leader_election.cc:288] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:11.916662 19527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:11.916893 19524 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:11.916909 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 19451
I20260811 11:20:11.916924 19527 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [term 1 LEADER]: Becoming Leader. State: Replica: f1b1c2a549b041f6b108b94ff3f32beb, State: Running, Role: LEADER
I20260811 11:20:11.917047 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:11.917018 19527 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b1c2a549b041f6b108b94ff3f32beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260811 11:20:11.917285 28772 external_mini_cluster.cc:1428] Setting key f4d9215c19c6e15182c0904e1307b852
I20260811 11:20:11.917434 19528 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1b1c2a549b041f6b108b94ff3f32beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b1c2a549b041f6b108b94ff3f32beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } } }
I20260811 11:20:11.917461 19529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1b1c2a549b041f6b108b94ff3f32beb. Latest consensus state: current_term: 1 leader_uuid: "f1b1c2a549b041f6b108b94ff3f32beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b1c2a549b041f6b108b94ff3f32beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } } }
I20260811 11:20:11.917577 19528 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:11.917589 19529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:11.917799 19532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:11.918351 19532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:11.920205 19532 catalog_manager.cc:1347] Generated new cluster ID: a6db26bf5e904fed858ce2842253322f
I20260811 11:20:11.920279 19532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:11.924937 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:11.918098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42933 (local address 127.0.0.1:41351)
0811 11:20:11.918777 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:11.918789 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:11.918826 (+ 37us) server_negotiation.cc:407] Connection header received
0811 11:20:11.919099 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:11.919107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:11.919175 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:11.919340 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:11.919651 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:11.921135 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:11.922372 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:11.923112 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:11.923236 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:11.924083 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:11.924088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:11.924088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:11.924319 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:11.924364 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:11.924381 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:11.924457 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:11.924542 (+ 85us) server_negotiation.cc:299] Negotiation successful
0811 11:20:11.924628 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":85,"threads_started":1}
I20260811 11:20:11.925982 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:11.929188 19532 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:11.929714 19532 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:11.932013 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:11.927994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34197 (local address 127.0.0.1:41351)
0811 11:20:11.928106 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:11.928108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:11.928191 (+ 83us) server_negotiation.cc:407] Connection header received
0811 11:20:11.928273 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:11.928274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:11.928286 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:11.928360 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:11.928511 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:11.929268 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:11.930382 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:11.930688 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:11.930737 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:11.931692 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:11.931693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:11.931693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:11.931699 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:11.931715 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:11.931720 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:11.931748 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:11.931869 (+ 121us) server_negotiation.cc:299] Negotiation successful
0811 11:20:11.931920 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260811 11:20:11.935037 19532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1b1c2a549b041f6b108b94ff3f32beb: Generated new TSK 0
I20260811 11:20:11.935245 19532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:11.941776 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 19451
2026-08-11T11:20:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:12 dist-test-slave-2w3w krb5kdc[19558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:12 dist-test-slave-2w3w krb5kdc[19558](info): set up 2 sockets
Aug 11 11:20:12 dist-test-slave-2w3w krb5kdc[19558](info): commencing operation
krb5kdc: starting...
W20260811 11:20:13.073418 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:13 dist-test-slave-2w3w krb5kdc[19558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:13.215440 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:13.305713 19571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:13.306033 19571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:13.306092 19571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:13.307456 19571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:13.307564 19571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:13.307590 19571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:13.307612 19571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:13.307643 19571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:13.309348 19571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:13.309700 19571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:13.309979 19571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:13.315826 19576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:13.315826 19580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:13.316092 19571 server_base.cc:1034] running on GCE node
W20260811 11:20:13.315843 19578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:13.316529 19571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:13.316892 19571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:13.318063 19571 hybrid_clock.cc:648] HybridClock initialized: now 1786447213318031 us; error 49 us; skew 500 ppm
Aug 11 11:20:13 dist-test-slave-2w3w krb5kdc[19558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:13.320991 19571 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:13.321591 19571 webserver.cc:460] Webserver started at http://127.0.0.1:43049/ using document root <none> and password file <none>
I20260811 11:20:13.321849 19571 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:13.321928 19571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:13.322093 19571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:13.323342 19571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "56220200ef824baf862eca675de60c7a"
format_stamp: "Formatted at 2026-08-11 11:20:13 on dist-test-slave-2w3w"
server_key: "770d664326bc1614a7c43aca42655948"
server_key_iv: "8fa7d0ad45b81bd022524b7f0d71c759"
server_key_version: "encryptionkey@0"
I20260811 11:20:13.323734 19571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "56220200ef824baf862eca675de60c7a"
format_stamp: "Formatted at 2026-08-11 11:20:13 on dist-test-slave-2w3w"
server_key: "770d664326bc1614a7c43aca42655948"
server_key_iv: "8fa7d0ad45b81bd022524b7f0d71c759"
server_key_version: "encryptionkey@0"
I20260811 11:20:13.325253 19571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:20:13.326175 19589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:13.326437 19571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:13.326545 19571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "56220200ef824baf862eca675de60c7a"
format_stamp: "Formatted at 2026-08-11 11:20:13 on dist-test-slave-2w3w"
server_key: "770d664326bc1614a7c43aca42655948"
server_key_iv: "8fa7d0ad45b81bd022524b7f0d71c759"
server_key_version: "encryptionkey@0"
I20260811 11:20:13.326610 19571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:13.330313 19571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:13.332864 19571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:13.332990 19571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:13.337311 19571 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41971
I20260811 11:20:13.337379 19650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41971 every 8 connection(s)
I20260811 11:20:13.337779 19571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:13.338954 19652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:13.341758 19652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a: Bootstrap starting.
I20260811 11:20:13.342005 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 19571
I20260811 11:20:13.342149 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:13.342352 28772 external_mini_cluster.cc:1428] Setting key 5d274c690c963c3e8dee10e0684f7362
I20260811 11:20:13.342535 19652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:13.342922 19652 log.cc:824] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:13.343755 19652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a: No bootstrap required, opened a new log
I20260811 11:20:13.345216 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:13.342932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37599 (local address 127.0.0.1:41971)
0811 11:20:13.343205 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:13.343216 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:13.343262 (+ 46us) server_negotiation.cc:407] Connection header received
0811 11:20:13.343809 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:13.343816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:13.343873 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:13.344014 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:13.344728 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:13.344731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:13.344731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:13.344735 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:13.344783 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:13.344795 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:13.344867 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:13.344948 (+ 81us) server_negotiation.cc:299] Negotiation successful
0811 11:20:13.345008 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":139,"threads_started":1}
I20260811 11:20:13.346313 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:13.346387 19652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56220200ef824baf862eca675de60c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41971 } }
I20260811 11:20:13.346560 19652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:13.346590 19652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56220200ef824baf862eca675de60c7a, State: Initialized, Role: FOLLOWER
I20260811 11:20:13.346707 19652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56220200ef824baf862eca675de60c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41971 } }
I20260811 11:20:13.346794 19652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:13.346841 19652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:13.346894 19652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:13.347904 19652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56220200ef824baf862eca675de60c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41971 } }
I20260811 11:20:13.348022 19652 leader_election.cc:302] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56220200ef824baf862eca675de60c7a; no voters:
I20260811 11:20:13.348213 19652 leader_election.cc:288] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:13.348316 19662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:13.348484 19662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [term 1 LEADER]: Becoming Leader. State: Replica: 56220200ef824baf862eca675de60c7a, State: Running, Role: LEADER
I20260811 11:20:13.348513 19652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:13.348598 19662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56220200ef824baf862eca675de60c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41971 } }
I20260811 11:20:13.349536 19665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56220200ef824baf862eca675de60c7a. Latest consensus state: current_term: 1 leader_uuid: "56220200ef824baf862eca675de60c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56220200ef824baf862eca675de60c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41971 } } }
I20260811 11:20:13.349642 19665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:13.349822 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:13.348153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51353 (local address 127.0.0.1:41971)
0811 11:20:13.348196 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:13.348198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:13.348429 (+ 231us) server_negotiation.cc:407] Connection header received
0811 11:20:13.348557 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:13.348558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:13.348564 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:13.348595 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:13.349415 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:13.349416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:13.349417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:13.349423 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:13.349439 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:13.349444 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:13.349493 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:13.349734 (+ 241us) server_negotiation.cc:299] Negotiation successful
0811 11:20:13.349750 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 11:20:13.350044 19664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56220200ef824baf862eca675de60c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56220200ef824baf862eca675de60c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41971 } } }
I20260811 11:20:13.350122 19664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:13.350060 19672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:13.350766 19672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:13.352684 19672 catalog_manager.cc:1347] Generated new cluster ID: f6e95f72d09849e78a30f6fc34e415a4
I20260811 11:20:13.352744 19672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:13.363222 19672 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:13.363466 19672 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:13.374383 19672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56220200ef824baf862eca675de60c7a: Generated new TSK 0
I20260811 11:20:13.374629 19672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:13.385129 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 19571
2026-08-11T11:20:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:13 dist-test-slave-2w3w krb5kdc[19685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:13 dist-test-slave-2w3w krb5kdc[19685](info): set up 2 sockets
Aug 11 11:20:13 dist-test-slave-2w3w krb5kdc[19685](info): commencing operation
krb5kdc: starting...
W20260811 11:20:14.504786 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:14 dist-test-slave-2w3w krb5kdc[19685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:14.650501 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:14.737440 19700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:14.737676 19700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:14.737702 19700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:14.739008 19700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:14.739063 19700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:14.739077 19700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:14.739090 19700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:14.739102 19700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:14.740780 19700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:14.741062 19700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:14.741371 19700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:14.746939 19707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:14.746991 19700 server_base.cc:1034] running on GCE node
W20260811 11:20:14.746946 19706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:14.746946 19710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:14.747488 19700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:14.747843 19700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:14.749043 19700 hybrid_clock.cc:648] HybridClock initialized: now 1786447214749002 us; error 54 us; skew 500 ppm
Aug 11 11:20:14 dist-test-slave-2w3w krb5kdc[19685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:14.752079 19700 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:14.752584 19700 webserver.cc:460] Webserver started at http://127.0.0.1:39159/ using document root <none> and password file <none>
I20260811 11:20:14.752861 19700 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:14.752936 19700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:14.753108 19700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:14.754246 19700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "f6cac0f65bdc4e6e817552de19184960"
format_stamp: "Formatted at 2026-08-11 11:20:14 on dist-test-slave-2w3w"
server_key: "5e012fbba6ee3603e409776249b83379"
server_key_iv: "d7ffa05017c7ccea7fd20261abcae271"
server_key_version: "encryptionkey@0"
I20260811 11:20:14.754633 19700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "f6cac0f65bdc4e6e817552de19184960"
format_stamp: "Formatted at 2026-08-11 11:20:14 on dist-test-slave-2w3w"
server_key: "5e012fbba6ee3603e409776249b83379"
server_key_iv: "d7ffa05017c7ccea7fd20261abcae271"
server_key_version: "encryptionkey@0"
I20260811 11:20:14.756250 19700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:20:14.757155 19718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:14.757392 19700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:14.757512 19700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "f6cac0f65bdc4e6e817552de19184960"
format_stamp: "Formatted at 2026-08-11 11:20:14 on dist-test-slave-2w3w"
server_key: "5e012fbba6ee3603e409776249b83379"
server_key_iv: "d7ffa05017c7ccea7fd20261abcae271"
server_key_version: "encryptionkey@0"
I20260811 11:20:14.757597 19700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:14.789083 19700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:14.791872 19700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:14.792006 19700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:14.796736 19700 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43039
I20260811 11:20:14.796790 19777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43039 every 8 connection(s)
I20260811 11:20:14.797212 19700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:14.797629 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 19700
I20260811 11:20:14.797789 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:14.798032 28772 external_mini_cluster.cc:1428] Setting key 742b05918cc41c29ce235d4863921953
I20260811 11:20:14.798460 19778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:14.802385 19778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960: Bootstrap starting.
I20260811 11:20:14.803200 19778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:14.803609 19778 log.cc:824] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:14.804503 19778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960: No bootstrap required, opened a new log
Aug 11 11:20:14 dist-test-slave-2w3w krb5kdc[19685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 11:20:14.807102 19778 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cac0f65bdc4e6e817552de19184960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43039 } }
I20260811 11:20:14.807329 19778 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:14.807381 19778 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6cac0f65bdc4e6e817552de19184960, State: Initialized, Role: FOLLOWER
I20260811 11:20:14.807569 19778 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cac0f65bdc4e6e817552de19184960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43039 } }
I20260811 11:20:14.807644 19778 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:14.807693 19778 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:14.807782 19778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:14.808573 19778 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cac0f65bdc4e6e817552de19184960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43039 } }
I20260811 11:20:14.808668 19778 leader_election.cc:302] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6cac0f65bdc4e6e817552de19184960; no voters:
I20260811 11:20:14.808959 19778 leader_election.cc:288] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:14.808992 19783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:14.809113 19783 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [term 1 LEADER]: Becoming Leader. State: Replica: f6cac0f65bdc4e6e817552de19184960, State: Running, Role: LEADER
I20260811 11:20:14.809264 19783 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cac0f65bdc4e6e817552de19184960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43039 } }
I20260811 11:20:14.809293 19778 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:14.809700 19785 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6cac0f65bdc4e6e817552de19184960. Latest consensus state: current_term: 1 leader_uuid: "f6cac0f65bdc4e6e817552de19184960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cac0f65bdc4e6e817552de19184960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43039 } } }
I20260811 11:20:14.809787 19785 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:14.809985 19784 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6cac0f65bdc4e6e817552de19184960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cac0f65bdc4e6e817552de19184960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43039 } } }
I20260811 11:20:14.810132 19784 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960 [sys.catalog]: This master's current role is: LEADER
W20260811 11:20:14.811520 19801 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260811 11:20:14.811539 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:14.798664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:56665 (local address 127.0.0.1:43039)
0811 11:20:14.798864 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:14.798874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:14.798911 (+ 37us) server_negotiation.cc:407] Connection header received
0811 11:20:14.799453 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:14.799461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:14.799516 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:14.799647 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:14.800556 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:14.801605 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:14.802719 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:14.803366 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:14.803470 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:14.806782 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:14.806787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:14.806788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:14.806802 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:14.809164 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:14.809776 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:14.809779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:14.809780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:14.809918 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:14.810223 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:14.810224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:14.810224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:14.810578 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:14.810747 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:14.811063 (+ 316us) server_negotiation.cc:299] Negotiation successful
0811 11:20:14.811186 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
W20260811 11:20:14.811618 19801 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:14.811685 19787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:14.811843 19787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:14.812791 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:14.813691 19787 catalog_manager.cc:1347] Generated new cluster ID: c5a1416281974f2c9dfd7b45f6c73541
I20260811 11:20:14.813759 19787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:14.826495 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:14.815433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:41669 (local address 127.0.0.1:43039)
0811 11:20:14.815488 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:14.815490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:14.819326 (+ 3836us) server_negotiation.cc:407] Connection header received
0811 11:20:14.819462 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:14.819464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:14.819477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:14.819561 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:14.820261 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:14.820847 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:14.821692 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:14.821973 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:14.822057 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:14.823585 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:14.823587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:14.823587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:14.823596 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:14.825169 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:14.825581 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:14.825582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:14.825583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:14.825648 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:14.825898 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:14.825899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:14.825899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:14.826027 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:14.826149 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:14.826290 (+ 141us) server_negotiation.cc:299] Negotiation successful
0811 11:20:14.826369 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:20:14.829043 19787 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:14.829691 19787 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:14.839473 19787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6cac0f65bdc4e6e817552de19184960: Generated new TSK 0
I20260811 11:20:14.839906 19787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:14.842370 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 19700
W20260811 11:20:14.851984 19682 connection.cc:535] client connection to 127.0.0.1:43039 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43039): Connection reset by peer (error 104)
2026-08-11T11:20:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:14 dist-test-slave-2w3w krb5kdc[19811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:14 dist-test-slave-2w3w krb5kdc[19811](info): set up 2 sockets
Aug 11 11:20:14 dist-test-slave-2w3w krb5kdc[19811](info): commencing operation
krb5kdc: starting...
W20260811 11:20:15.969715 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:16 dist-test-slave-2w3w krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:16.111019 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:16.197343 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:16.197631 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:16.197678 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:16.199019 19828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:16.199105 19828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:16.199129 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:16.199151 19828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:16.199182 19828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:16.200886 19828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:16.201227 19828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:16.201514 19828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:16.207021 19837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:16.207158 19828 server_base.cc:1034] running on GCE node
W20260811 11:20:16.207003 19835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:16.207274 19833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:16.207602 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:16.207957 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:16.209149 19828 hybrid_clock.cc:648] HybridClock initialized: now 1786447216209134 us; error 47 us; skew 500 ppm
Aug 11 11:20:16 dist-test-slave-2w3w krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:16.211970 19828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:16.212492 19828 webserver.cc:460] Webserver started at http://127.0.0.1:37273/ using document root <none> and password file <none>
I20260811 11:20:16.212761 19828 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:16.212836 19828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:16.212997 19828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:16.214167 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "bb2f5605eed94037b38111b2876478d7"
format_stamp: "Formatted at 2026-08-11 11:20:16 on dist-test-slave-2w3w"
server_key: "6f2e61b0ee6865903095bd717df6f5fc"
server_key_iv: "58695aedd7a0731eb32420fb382b9bfe"
server_key_version: "encryptionkey@0"
I20260811 11:20:16.214560 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "bb2f5605eed94037b38111b2876478d7"
format_stamp: "Formatted at 2026-08-11 11:20:16 on dist-test-slave-2w3w"
server_key: "6f2e61b0ee6865903095bd717df6f5fc"
server_key_iv: "58695aedd7a0731eb32420fb382b9bfe"
server_key_version: "encryptionkey@0"
I20260811 11:20:16.216162 19828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:20:16.217139 19846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:16.217383 19828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:16.217531 19828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "bb2f5605eed94037b38111b2876478d7"
format_stamp: "Formatted at 2026-08-11 11:20:16 on dist-test-slave-2w3w"
server_key: "6f2e61b0ee6865903095bd717df6f5fc"
server_key_iv: "58695aedd7a0731eb32420fb382b9bfe"
server_key_version: "encryptionkey@0"
I20260811 11:20:16.217620 19828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:16.233455 19828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:16.236387 19828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:16.236605 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:16.240867 19828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32769
I20260811 11:20:16.240932 19906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32769 every 8 connection(s)
I20260811 11:20:16.241328 19828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:16.242326 19907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:16.244885 19907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7: Bootstrap starting.
I20260811 11:20:16.245469 19907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:16.245735 19907 log.cc:824] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:16.246510 19907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7: No bootstrap required, opened a new log
I20260811 11:20:16.247771 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 19828
I20260811 11:20:16.247902 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:16.248099 28772 external_mini_cluster.cc:1428] Setting key 45044b9ac4424fba1abf975b57dcdfd6
I20260811 11:20:16.248852 19907 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2f5605eed94037b38111b2876478d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260811 11:20:16.249082 19907 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:16.249141 19907 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2f5605eed94037b38111b2876478d7, State: Initialized, Role: FOLLOWER
I20260811 11:20:16.249282 19907 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2f5605eed94037b38111b2876478d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260811 11:20:16.249365 19907 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:16.249389 19907 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:16.249449 19907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:16.250676 19907 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2f5605eed94037b38111b2876478d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260811 11:20:16.250787 19907 leader_election.cc:302] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb2f5605eed94037b38111b2876478d7; no voters:
I20260811 11:20:16.250978 19907 leader_election.cc:288] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:16.251022 19912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:16.251204 19912 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [term 1 LEADER]: Becoming Leader. State: Replica: bb2f5605eed94037b38111b2876478d7, State: Running, Role: LEADER
I20260811 11:20:16.251242 19907 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:16.251307 19912 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2f5605eed94037b38111b2876478d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260811 11:20:16.251652 19914 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb2f5605eed94037b38111b2876478d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2f5605eed94037b38111b2876478d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } } }
I20260811 11:20:16.251631 19912 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb2f5605eed94037b38111b2876478d7. Latest consensus state: current_term: 1 leader_uuid: "bb2f5605eed94037b38111b2876478d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2f5605eed94037b38111b2876478d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } } }
I20260811 11:20:16.251787 19912 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:16.251776 19914 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:16.252648 19920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:16.253185 19920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:16.255034 19920 catalog_manager.cc:1347] Generated new cluster ID: 84739e17a5b64dfab79e6d8093b582ad
I20260811 11:20:16.255113 19920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:16.255223 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:16.248814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:57453 (local address 127.0.0.1:32769)
0811 11:20:16.249059 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:16.249069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:16.249107 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:20:16.249241 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:16.249248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:16.249310 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:16.249480 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:16.249841 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:16.251346 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:16.252621 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:16.253298 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:16.253430 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:16.254278 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:16.254282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:16.254282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:16.254532 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:16.254578 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:16.254593 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:16.254679 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:16.254875 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 11:20:16.254907 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":119,"threads_started":1}
I20260811 11:20:16.256209 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:16.262694 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:16.257645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:33353 (local address 127.0.0.1:32769)
0811 11:20:16.257684 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:16.257686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:16.259210 (+ 1524us) server_negotiation.cc:407] Connection header received
0811 11:20:16.259321 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:16.259322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:16.259334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:16.259407 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:16.259562 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:16.260527 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:16.261319 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:16.261571 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:16.261651 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:16.262396 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:16.262397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:16.262397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:16.262402 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:16.262417 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:16.262421 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:16.262492 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:16.262570 (+ 78us) server_negotiation.cc:299] Negotiation successful
0811 11:20:16.262585 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 11:20:16.275831 19920 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:16.276472 19920 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:16.280268 19920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb2f5605eed94037b38111b2876478d7: Generated new TSK 0
I20260811 11:20:16.280531 19920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:16.288368 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 19828
W20260811 11:20:16.296532 19809 connection.cc:535] client connection to 127.0.0.1:32769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32769): Connection reset by peer (error 104)
2026-08-11T11:20:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:16 dist-test-slave-2w3w krb5kdc[19942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:16 dist-test-slave-2w3w krb5kdc[19942](info): set up 2 sockets
Aug 11 11:20:16 dist-test-slave-2w3w krb5kdc[19942](info): commencing operation
krb5kdc: starting...
W20260811 11:20:17.409963 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:17 dist-test-slave-2w3w krb5kdc[19942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:17.549888 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:17.642599 19957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:17.642836 19957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:17.642861 19957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:17.644060 19957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:17.644107 19957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:17.644122 19957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:17.644134 19957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:17.644146 19957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:17.645599 19957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:17.645809 19957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:17.646042 19957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:17.651006 19965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:17.651010 19962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:17.651012 19963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:17.651340 19957 server_base.cc:1034] running on GCE node
I20260811 11:20:17.651587 19957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:17.651846 19957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:17.653031 19957 hybrid_clock.cc:648] HybridClock initialized: now 1786447217653022 us; error 34 us; skew 500 ppm
Aug 11 11:20:17 dist-test-slave-2w3w krb5kdc[19942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:17.661935 19957 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:17.662515 19957 webserver.cc:460] Webserver started at http://127.0.0.1:34281/ using document root <none> and password file <none>
I20260811 11:20:17.662815 19957 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:17.662895 19957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:17.663098 19957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:17.664335 19957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "8a887220ae2a47a89c3410495dd58939"
format_stamp: "Formatted at 2026-08-11 11:20:17 on dist-test-slave-2w3w"
server_key: "db7d99184df739502f15ad8b76853e55"
server_key_iv: "fd536e2d41a2e7feed517f82e94db621"
server_key_version: "encryptionkey@0"
I20260811 11:20:17.664743 19957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "8a887220ae2a47a89c3410495dd58939"
format_stamp: "Formatted at 2026-08-11 11:20:17 on dist-test-slave-2w3w"
server_key: "db7d99184df739502f15ad8b76853e55"
server_key_iv: "fd536e2d41a2e7feed517f82e94db621"
server_key_version: "encryptionkey@0"
I20260811 11:20:17.666404 19957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:20:17.667382 19975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:17.667613 19957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:17.667729 19957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "8a887220ae2a47a89c3410495dd58939"
format_stamp: "Formatted at 2026-08-11 11:20:17 on dist-test-slave-2w3w"
server_key: "db7d99184df739502f15ad8b76853e55"
server_key_iv: "fd536e2d41a2e7feed517f82e94db621"
server_key_version: "encryptionkey@0"
I20260811 11:20:17.667850 19957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:17.671799 19957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:17.674319 19957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:17.674521 19957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:17.679582 19957 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43301
I20260811 11:20:17.679616 20034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43301 every 8 connection(s)
I20260811 11:20:17.680020 19957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:17.681113 20035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:17.683710 20035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939: Bootstrap starting.
I20260811 11:20:17.684545 20035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:17.684976 20035 log.cc:824] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:17.685855 20035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939: No bootstrap required, opened a new log
I20260811 11:20:17.686054 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 19957
I20260811 11:20:17.686208 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:17.686440 28772 external_mini_cluster.cc:1428] Setting key f157b33267dd137a053f87a15caf147f
I20260811 11:20:17.688519 20035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a887220ae2a47a89c3410495dd58939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43301 } }
I20260811 11:20:17.688745 20035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:17.688782 20035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a887220ae2a47a89c3410495dd58939, State: Initialized, Role: FOLLOWER
I20260811 11:20:17.688921 20035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a887220ae2a47a89c3410495dd58939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43301 } }
I20260811 11:20:17.689008 20035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:17.689047 20035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:17.689148 20035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:17.689769 20035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a887220ae2a47a89c3410495dd58939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43301 } }
I20260811 11:20:17.689918 20035 leader_election.cc:302] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a887220ae2a47a89c3410495dd58939; no voters:
I20260811 11:20:17.690096 20035 leader_election.cc:288] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:17.690176 20040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:17.690397 20040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [term 1 LEADER]: Becoming Leader. State: Replica: 8a887220ae2a47a89c3410495dd58939, State: Running, Role: LEADER
I20260811 11:20:17.690457 20035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:17.690490 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:17.687092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:58341 (local address 127.0.0.1:43301)
0811 11:20:17.687305 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:17.687315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:17.687384 (+ 69us) server_negotiation.cc:407] Connection header received
0811 11:20:17.688730 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:17.688739 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:17.688778 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:17.688946 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:17.689863 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:17.689867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:17.689867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:17.689872 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:17.689921 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:17.689938 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:17.690014 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:17.690161 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 11:20:17.690248 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260811 11:20:17.690527 20040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a887220ae2a47a89c3410495dd58939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43301 } }
I20260811 11:20:17.691084 20041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a887220ae2a47a89c3410495dd58939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a887220ae2a47a89c3410495dd58939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43301 } } }
I20260811 11:20:17.691175 20041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:17.691944 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:17.691915 20042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a887220ae2a47a89c3410495dd58939. Latest consensus state: current_term: 1 leader_uuid: "8a887220ae2a47a89c3410495dd58939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a887220ae2a47a89c3410495dd58939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43301 } } }
W20260811 11:20:17.692484 20058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260811 11:20:17.693598 20042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939 [sys.catalog]: This master's current role is: LEADER
W20260811 11:20:17.693609 20058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:17.693706 20050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:17.693919 20050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:17.695626 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:17.693672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:39465 (local address 127.0.0.1:43301)
0811 11:20:17.693714 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:17.693716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:17.693851 (+ 135us) server_negotiation.cc:407] Connection header received
0811 11:20:17.693939 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:17.693940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:17.693945 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:17.694007 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:17.695152 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:17.695153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:17.695153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:17.695158 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:17.695173 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:17.695178 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:17.695258 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:17.695543 (+ 285us) server_negotiation.cc:299] Negotiation successful
0811 11:20:17.695561 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 11:20:17.696024 20050 catalog_manager.cc:1347] Generated new cluster ID: ade41c11e08542959426cae141142f55
I20260811 11:20:17.696074 20050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:17.704977 20050 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:17.705179 20050 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:17.712155 20050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a887220ae2a47a89c3410495dd58939: Generated new TSK 0
I20260811 11:20:17.712404 20050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:17.715929 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 19957
2026-08-11T11:20:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:17 dist-test-slave-2w3w krb5kdc[20068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:17 dist-test-slave-2w3w krb5kdc[20068](info): set up 2 sockets
Aug 11 11:20:17 dist-test-slave-2w3w krb5kdc[20068](info): commencing operation
krb5kdc: starting...
W20260811 11:20:18.832460 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:18 dist-test-slave-2w3w krb5kdc[20068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:18.974907 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:19.064025 20081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:19.064245 20081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:19.064270 20081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:19.065475 20081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:19.065521 20081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:19.065536 20081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:19.065577 20081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:19.065604 20081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:19.067250 20081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:19.067521 20081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:19.067765 20081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:19.073066 20086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:19.073055 20087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:19.073060 20089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:19.073259 20081 server_base.cc:1034] running on GCE node
I20260811 11:20:19.073653 20081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:19.074033 20081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:19.075204 20081 hybrid_clock.cc:648] HybridClock initialized: now 1786447219075177 us; error 60 us; skew 500 ppm
Aug 11 11:20:19 dist-test-slave-2w3w krb5kdc[20068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:19.077978 20081 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:19.078418 20081 webserver.cc:460] Webserver started at http://127.0.0.1:42585/ using document root <none> and password file <none>
I20260811 11:20:19.078616 20081 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:19.078658 20081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:19.078751 20081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:19.079897 20081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "6bff68d1f4364262953712bf094b7563"
format_stamp: "Formatted at 2026-08-11 11:20:19 on dist-test-slave-2w3w"
server_key: "227c6136bafd79a679672faa57c7ca5e"
server_key_iv: "dbcc9efe74b94051c69574d6526311b2"
server_key_version: "encryptionkey@0"
I20260811 11:20:19.080220 20081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "6bff68d1f4364262953712bf094b7563"
format_stamp: "Formatted at 2026-08-11 11:20:19 on dist-test-slave-2w3w"
server_key: "227c6136bafd79a679672faa57c7ca5e"
server_key_iv: "dbcc9efe74b94051c69574d6526311b2"
server_key_version: "encryptionkey@0"
I20260811 11:20:19.081708 20081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 11:20:19.082644 20096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:19.082892 20081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:19.083032 20081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "6bff68d1f4364262953712bf094b7563"
format_stamp: "Formatted at 2026-08-11 11:20:19 on dist-test-slave-2w3w"
server_key: "227c6136bafd79a679672faa57c7ca5e"
server_key_iv: "dbcc9efe74b94051c69574d6526311b2"
server_key_version: "encryptionkey@0"
I20260811 11:20:19.083123 20081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:19.090000 20081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:19.092484 20081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:19.092671 20081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:19.097292 20081 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35527
I20260811 11:20:19.097386 20155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35527 every 8 connection(s)
I20260811 11:20:19.097769 20081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:19.098841 20156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:19.101414 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 20081
I20260811 11:20:19.101470 20156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563: Bootstrap starting.
I20260811 11:20:19.101550 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:19.101759 28772 external_mini_cluster.cc:1428] Setting key 08564b1c90d7538c534d05807dede074
I20260811 11:20:19.102324 20156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:19.102679 20156 log.cc:824] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:19.103632 20156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563: No bootstrap required, opened a new log
I20260811 11:20:19.105997 20156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff68d1f4364262953712bf094b7563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260811 11:20:19.106185 20156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:19.106225 20156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff68d1f4364262953712bf094b7563, State: Initialized, Role: FOLLOWER
I20260811 11:20:19.106396 20156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff68d1f4364262953712bf094b7563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260811 11:20:19.106528 20156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:19.106577 20156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:19.106657 20156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:19.107547 20156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff68d1f4364262953712bf094b7563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260811 11:20:19.107657 20156 leader_election.cc:302] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bff68d1f4364262953712bf094b7563; no voters:
I20260811 11:20:19.107869 20156 leader_election.cc:288] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:19.107940 20165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:19.108103 20165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [term 1 LEADER]: Becoming Leader. State: Replica: 6bff68d1f4364262953712bf094b7563, State: Running, Role: LEADER
I20260811 11:20:19.108180 20156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:19.108243 20165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff68d1f4364262953712bf094b7563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260811 11:20:19.108606 20167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bff68d1f4364262953712bf094b7563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff68d1f4364262953712bf094b7563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } } }
I20260811 11:20:19.108595 20165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bff68d1f4364262953712bf094b7563. Latest consensus state: current_term: 1 leader_uuid: "6bff68d1f4364262953712bf094b7563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff68d1f4364262953712bf094b7563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } } }
I20260811 11:20:19.108707 20167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:19.108711 20165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:19.109273 20173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:19.109891 20173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 11 11:20:19 dist-test-slave-2w3w krb5kdc[20068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 11:20:19.111634 20173 catalog_manager.cc:1347] Generated new cluster ID: 8a814c9facc34a5c96609ee83190dbc7
I20260811 11:20:19.111688 20173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:19.114432 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:19.102349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:36269 (local address 127.0.0.1:35527)
0811 11:20:19.102544 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:19.102552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:19.102579 (+ 27us) server_negotiation.cc:407] Connection header received
0811 11:20:19.103281 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:19.103295 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:19.103353 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:19.103478 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:19.104359 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:19.105301 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:19.106478 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:19.107102 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:19.107197 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:19.110539 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:19.110544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:19.110544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:19.110559 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:19.112620 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:19.113116 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:19.113120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:19.113120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:19.113216 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:19.113467 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:19.113468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:19.113468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:19.113715 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:19.113950 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:19.114090 (+ 140us) server_negotiation.cc:299] Negotiation successful
0811 11:20:19.114225 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260811 11:20:19.115577 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:19.125852 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:19.117019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:42649 (local address 127.0.0.1:35527)
0811 11:20:19.117059 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:19.117061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:19.119185 (+ 2124us) server_negotiation.cc:407] Connection header received
0811 11:20:19.119261 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:19.119261 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:19.119273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:19.119323 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:19.119975 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:19.120526 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:19.121274 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:19.121505 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:19.121587 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:19.122900 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:19.122901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:19.122902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:19.122909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:19.124550 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:19.124965 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:19.124967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:19.124967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:19.125034 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:19.125273 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:19.125275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:19.125275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:19.125403 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:19.125550 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:19.125668 (+ 118us) server_negotiation.cc:299] Negotiation successful
0811 11:20:19.125734 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 11:20:19.129665 20173 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:19.130434 20173 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:19.144465 20173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bff68d1f4364262953712bf094b7563: Generated new TSK 0
I20260811 11:20:19.144802 20173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:19.154695 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 20081
W20260811 11:20:19.162180 20066 connection.cc:535] client connection to 127.0.0.1:35527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35527): Connection reset by peer (error 104)
2026-08-11T11:20:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:19 dist-test-slave-2w3w krb5kdc[20194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:19 dist-test-slave-2w3w krb5kdc[20194](info): set up 2 sockets
Aug 11 11:20:19 dist-test-slave-2w3w krb5kdc[20194](info): commencing operation
krb5kdc: starting...
W20260811 11:20:20.292991 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:20 dist-test-slave-2w3w krb5kdc[20194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:20.435875 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:20.522104 20208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:20.522388 20208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:20.522434 20208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:20.523770 20208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:20.523864 20208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:20.523891 20208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:20.523916 20208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:20.523945 20208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:20.525590 20208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:20.525915 20208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:20.526185 20208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:20.531450 20213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:20.531450 20216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:20.531481 20214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:20.531934 20208 server_base.cc:1034] running on GCE node
I20260811 11:20:20.532191 20208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:20.532518 20208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:20.533703 20208 hybrid_clock.cc:648] HybridClock initialized: now 1786447220533690 us; error 33 us; skew 500 ppm
Aug 11 11:20:20 dist-test-slave-2w3w krb5kdc[20194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:20.536509 20208 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:20.536957 20208 webserver.cc:460] Webserver started at http://127.0.0.1:36991/ using document root <none> and password file <none>
I20260811 11:20:20.537220 20208 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:20.537267 20208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:20.537442 20208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:20.538630 20208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "5220c56a081645b088121efb88c23ccc"
format_stamp: "Formatted at 2026-08-11 11:20:20 on dist-test-slave-2w3w"
server_key: "60dbd142cf79a90194f8477041e59e8b"
server_key_iv: "4a471ba7e6d320ced26b97b6bf5e2e26"
server_key_version: "encryptionkey@0"
I20260811 11:20:20.539083 20208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "5220c56a081645b088121efb88c23ccc"
format_stamp: "Formatted at 2026-08-11 11:20:20 on dist-test-slave-2w3w"
server_key: "60dbd142cf79a90194f8477041e59e8b"
server_key_iv: "4a471ba7e6d320ced26b97b6bf5e2e26"
server_key_version: "encryptionkey@0"
I20260811 11:20:20.540604 20208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:20.541560 20223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:20.541849 20208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:20.541985 20208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "5220c56a081645b088121efb88c23ccc"
format_stamp: "Formatted at 2026-08-11 11:20:20 on dist-test-slave-2w3w"
server_key: "60dbd142cf79a90194f8477041e59e8b"
server_key_iv: "4a471ba7e6d320ced26b97b6bf5e2e26"
server_key_version: "encryptionkey@0"
I20260811 11:20:20.542078 20208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:20.550137 20208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:20.552688 20208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:20.552866 20208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:20.557309 20208 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42811
I20260811 11:20:20.557348 20280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42811 every 8 connection(s)
I20260811 11:20:20.557742 20208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:20.558661 20281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:20.561158 20281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc: Bootstrap starting.
I20260811 11:20:20.561796 20281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:20.562117 20281 log.cc:824] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:20.562551 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 20208
I20260811 11:20:20.562700 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:20.563030 28772 external_mini_cluster.cc:1428] Setting key 4af1fb68e553832bbed26d5a6bcfb4a1
I20260811 11:20:20.563117 20281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc: No bootstrap required, opened a new log
I20260811 11:20:20.565056 20281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5220c56a081645b088121efb88c23ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260811 11:20:20.565200 20281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:20.565223 20281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5220c56a081645b088121efb88c23ccc, State: Initialized, Role: FOLLOWER
I20260811 11:20:20.565361 20281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5220c56a081645b088121efb88c23ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260811 11:20:20.565459 20281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:20.565506 20281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:20.565574 20281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:20.566246 20281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5220c56a081645b088121efb88c23ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260811 11:20:20.566376 20281 leader_election.cc:302] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5220c56a081645b088121efb88c23ccc; no voters:
I20260811 11:20:20.566563 20281 leader_election.cc:288] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:20.566614 20287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:20.566776 20287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [term 1 LEADER]: Becoming Leader. State: Replica: 5220c56a081645b088121efb88c23ccc, State: Running, Role: LEADER
I20260811 11:20:20.566834 20281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:20.566979 20287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5220c56a081645b088121efb88c23ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260811 11:20:20.567806 20289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5220c56a081645b088121efb88c23ccc. Latest consensus state: current_term: 1 leader_uuid: "5220c56a081645b088121efb88c23ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5220c56a081645b088121efb88c23ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } } }
I20260811 11:20:20.567905 20289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:20.568277 20298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:20.568414 20288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5220c56a081645b088121efb88c23ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5220c56a081645b088121efb88c23ccc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } } }
I20260811 11:20:20.568711 20288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:20.568814 20298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:20.570572 20298 catalog_manager.cc:1347] Generated new cluster ID: 3900c86dc88546b280369ea24295d9a1
I20260811 11:20:20.570688 20298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:20.570986 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:20.563824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:59175 (local address 127.0.0.1:42811)
0811 11:20:20.564031 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:20.564041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:20.564078 (+ 37us) server_negotiation.cc:407] Connection header received
0811 11:20:20.565058 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:20.565067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:20.565127 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:20.565343 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:20.565603 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:20.567351 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:20.568525 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:20.569165 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:20.569301 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:20.570148 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:20.570153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:20.570153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:20.570361 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:20.570399 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:20.570411 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:20.570526 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:20.570695 (+ 169us) server_negotiation.cc:299] Negotiation successful
0811 11:20:20.570725 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":117,"threads_started":1}
I20260811 11:20:20.572001 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:20.577963 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:20.573759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:52899 (local address 127.0.0.1:42811)
0811 11:20:20.573793 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:20.573795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:20.573803 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:20:20.573867 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:20.573868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:20.573877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:20.573930 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:20.574083 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:20.574814 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:20.576101 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:20.576779 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:20.576854 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:20.577680 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:20.577682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:20.577682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:20.577688 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:20.577704 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:20.577709 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:20.577763 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:20.577860 (+ 97us) server_negotiation.cc:299] Negotiation successful
0811 11:20:20.577875 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260811 11:20:20.599910 20298 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:20.600800 20298 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:20.611963 20298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5220c56a081645b088121efb88c23ccc: Generated new TSK 0
I20260811 11:20:20.612241 20298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:20.616845 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 20208
W20260811 11:20:20.623611 20191 connection.cc:535] client connection to 127.0.0.1:42811 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42811): Connection reset by peer (error 104)
2026-08-11T11:20:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:20 dist-test-slave-2w3w krb5kdc[20317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:20 dist-test-slave-2w3w krb5kdc[20317](info): set up 2 sockets
Aug 11 11:20:20 dist-test-slave-2w3w krb5kdc[20317](info): commencing operation
krb5kdc: starting...
W20260811 11:20:21.740293 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:21 dist-test-slave-2w3w krb5kdc[20317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:20:21.887638 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:21.976629 20330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:21.976918 20330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:21.976964 20330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:21.978272 20330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:21.978358 20330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:21.978384 20330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:21.978417 20330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:21.978441 20330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:21.980232 20330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:21.980556 20330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:21.980813 20330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:21.986301 20338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:21.986397 20336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:21.986299 20335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:21.986603 20330 server_base.cc:1034] running on GCE node
I20260811 11:20:21.986918 20330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:21.987322 20330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:21.988485 20330 hybrid_clock.cc:648] HybridClock initialized: now 1786447221988459 us; error 40 us; skew 500 ppm
Aug 11 11:20:21 dist-test-slave-2w3w krb5kdc[20317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:21.991453 20330 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:20:21.992072 20330 webserver.cc:460] Webserver started at http://127.0.0.1:44707/ using document root <none> and password file <none>
I20260811 11:20:21.992337 20330 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:21.992390 20330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:21.992532 20330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:21.993688 20330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "127b450e428d42f58d6390547b71681f"
format_stamp: "Formatted at 2026-08-11 11:20:21 on dist-test-slave-2w3w"
server_key: "59344161e2fabfc2bb4d14eafaf47384"
server_key_iv: "944e545653c0ab17f57717d305ddcbdc"
server_key_version: "encryptionkey@0"
I20260811 11:20:21.994100 20330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "127b450e428d42f58d6390547b71681f"
format_stamp: "Formatted at 2026-08-11 11:20:21 on dist-test-slave-2w3w"
server_key: "59344161e2fabfc2bb4d14eafaf47384"
server_key_iv: "944e545653c0ab17f57717d305ddcbdc"
server_key_version: "encryptionkey@0"
I20260811 11:20:21.995723 20330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:20:21.996711 20345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:21.997087 20330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:21.997202 20330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "127b450e428d42f58d6390547b71681f"
format_stamp: "Formatted at 2026-08-11 11:20:21 on dist-test-slave-2w3w"
server_key: "59344161e2fabfc2bb4d14eafaf47384"
server_key_iv: "944e545653c0ab17f57717d305ddcbdc"
server_key_version: "encryptionkey@0"
I20260811 11:20:21.997377 20330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:22.001303 20330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:22.004050 20330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:22.004179 20330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:22.009171 20330 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34369
I20260811 11:20:22.009253 20400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34369 every 8 connection(s)
I20260811 11:20:22.009609 20330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:22.010778 20401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:22.013513 20401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f: Bootstrap starting.
I20260811 11:20:22.014201 20401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:22.014510 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 20330
I20260811 11:20:22.014603 20401 log.cc:824] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:22.014650 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:22.014884 28772 external_mini_cluster.cc:1428] Setting key 731e6b4bc8d095e891673ec0d0de59ae
I20260811 11:20:22.015718 20401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f: No bootstrap required, opened a new log
I20260811 11:20:22.018180 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:22.015536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:41839 (local address 127.0.0.1:34369)
0811 11:20:22.015784 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:22.015795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:22.015833 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:20:22.016355 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:22.016362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:22.016392 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:22.016572 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:22.017493 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:22.017500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:22.017511 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:22.017517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:22.017572 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:22.017585 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:22.017691 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:22.017736 (+ 45us) server_negotiation.cc:299] Negotiation successful
0811 11:20:22.017860 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
I20260811 11:20:22.018105 20401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127b450e428d42f58d6390547b71681f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34369 } }
I20260811 11:20:22.018262 20401 raft_consensus.cc:374] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:22.018332 20401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127b450e428d42f58d6390547b71681f, State: Initialized, Role: FOLLOWER
I20260811 11:20:22.018476 20401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127b450e428d42f58d6390547b71681f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34369 } }
I20260811 11:20:22.018610 20401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:22.018639 20401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:22.018713 20401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:22.019405 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:22.019685 20401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127b450e428d42f58d6390547b71681f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34369 } }
I20260811 11:20:22.019814 20401 leader_election.cc:302] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 127b450e428d42f58d6390547b71681f; no voters:
I20260811 11:20:22.020061 20401 leader_election.cc:288] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:22.020092 20406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:22.020228 20406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [term 1 LEADER]: Becoming Leader. State: Replica: 127b450e428d42f58d6390547b71681f, State: Running, Role: LEADER
I20260811 11:20:22.020375 20401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:22.020380 20406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127b450e428d42f58d6390547b71681f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34369 } }
I20260811 11:20:22.020838 20408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "127b450e428d42f58d6390547b71681f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127b450e428d42f58d6390547b71681f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34369 } } }
I20260811 11:20:22.020961 20408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:22.020816 20406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 127b450e428d42f58d6390547b71681f. Latest consensus state: current_term: 1 leader_uuid: "127b450e428d42f58d6390547b71681f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127b450e428d42f58d6390547b71681f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34369 } } }
I20260811 11:20:22.021230 20406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:22.021381 20421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:22.021939 20421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:22.024184 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:22.022106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:50745 (local address 127.0.0.1:34369)
0811 11:20:22.022147 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:22.022149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:22.023043 (+ 894us) server_negotiation.cc:407] Connection header received
0811 11:20:22.023076 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:22.023077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:22.023085 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:22.023143 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:22.023914 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:22.023916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:22.023916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:22.023921 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:22.023943 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:22.023948 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:22.024019 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:22.024085 (+ 66us) server_negotiation.cc:299] Negotiation successful
0811 11:20:22.024101 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260811 11:20:22.024345 20421 catalog_manager.cc:1347] Generated new cluster ID: 508ecdf8bb3a49edabb1999206f3b2ab
I20260811 11:20:22.024394 20421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:22.035046 20421 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:22.035327 20421 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:22.054432 20421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 127b450e428d42f58d6390547b71681f: Generated new TSK 0
I20260811 11:20:22.054700 20421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:22.061568 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 20330
2026-08-11T11:20:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:22 dist-test-slave-2w3w krb5kdc[20437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:22 dist-test-slave-2w3w krb5kdc[20437](info): set up 2 sockets
Aug 11 11:20:22 dist-test-slave-2w3w krb5kdc[20437](info): commencing operation
krb5kdc: starting...
W20260811 11:20:23.230995 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:23 dist-test-slave-2w3w krb5kdc[20437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:20:23.370836 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:41909
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:40805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:23.456040 20451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:23.456326 20451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:23.456369 20451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:23.457656 20451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:23.457744 20451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:23.457772 20451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:23.457826 20451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:23.457855 20451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:23.459618 20451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:40805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:41909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:23.459935 20451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:23.460191 20451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:23.465421 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:23.465433 20457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:23.465457 20461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:23.465584 20451 server_base.cc:1034] running on GCE node
I20260811 11:20:23.465967 20451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:23.466384 20451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:23.467610 20451 hybrid_clock.cc:648] HybridClock initialized: now 1786447223467598 us; error 32 us; skew 500 ppm
Aug 11 11:20:23 dist-test-slave-2w3w krb5kdc[20437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:23.476003 20451 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:20:23.476639 20451 webserver.cc:460] Webserver started at http://127.28.25.62:36933/ using document root <none> and password file <none>
I20260811 11:20:23.476977 20451 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:23.477046 20451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:23.477258 20451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:23.478500 20451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "21e8c0c723c8465cbf91446f88026c61"
format_stamp: "Formatted at 2026-08-11 11:20:23 on dist-test-slave-2w3w"
server_key: "61c1a7b5368cd0dd6b64c7c26b600103"
server_key_iv: "7d4d23513deb301ac89dba9e3ca10a92"
server_key_version: "encryptionkey@0"
I20260811 11:20:23.478932 20451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "21e8c0c723c8465cbf91446f88026c61"
format_stamp: "Formatted at 2026-08-11 11:20:23 on dist-test-slave-2w3w"
server_key: "61c1a7b5368cd0dd6b64c7c26b600103"
server_key_iv: "7d4d23513deb301ac89dba9e3ca10a92"
server_key_version: "encryptionkey@0"
I20260811 11:20:23.480682 20451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:23.481792 20469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:23.482232 20451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:23.482393 20451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "21e8c0c723c8465cbf91446f88026c61"
format_stamp: "Formatted at 2026-08-11 11:20:23 on dist-test-slave-2w3w"
server_key: "61c1a7b5368cd0dd6b64c7c26b600103"
server_key_iv: "7d4d23513deb301ac89dba9e3ca10a92"
server_key_version: "encryptionkey@0"
I20260811 11:20:23.482506 20451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:23.519394 20451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:23.522401 20451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:23.522676 20451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:23.528735 20451 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:41909
I20260811 11:20:23.528766 20528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:41909 every 8 connection(s)
I20260811 11:20:23.529376 20451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:23.530388 20529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:23.532918 20529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61: Bootstrap starting.
I20260811 11:20:23.533546 20529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:23.533859 20529 log.cc:824] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:23.534675 20529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61: No bootstrap required, opened a new log
I20260811 11:20:23.536459 20529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e8c0c723c8465cbf91446f88026c61" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41909 } }
I20260811 11:20:23.536609 20529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:23.536633 20529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e8c0c723c8465cbf91446f88026c61, State: Initialized, Role: FOLLOWER
I20260811 11:20:23.536805 20529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e8c0c723c8465cbf91446f88026c61" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41909 } }
I20260811 11:20:23.536873 20529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:23.536895 20529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:23.536924 20529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:23.537796 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 20451
I20260811 11:20:23.537855 20529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e8c0c723c8465cbf91446f88026c61" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41909 } }
I20260811 11:20:23.537954 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:23.538018 20529 leader_election.cc:302] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21e8c0c723c8465cbf91446f88026c61; no voters:
I20260811 11:20:23.538203 28772 external_mini_cluster.cc:1428] Setting key 4beb8d9f1ca6faf7414eede8414a2b29
I20260811 11:20:23.538244 20529 leader_election.cc:288] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:23.538276 20533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:23.538444 20533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [term 1 LEADER]: Becoming Leader. State: Replica: 21e8c0c723c8465cbf91446f88026c61, State: Running, Role: LEADER
I20260811 11:20:23.538612 20529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:23.538585 20533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e8c0c723c8465cbf91446f88026c61" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41909 } }
I20260811 11:20:23.539116 20534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21e8c0c723c8465cbf91446f88026c61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e8c0c723c8465cbf91446f88026c61" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41909 } } }
I20260811 11:20:23.539386 20534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:23.539304 20535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21e8c0c723c8465cbf91446f88026c61. Latest consensus state: current_term: 1 leader_uuid: "21e8c0c723c8465cbf91446f88026c61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e8c0c723c8465cbf91446f88026c61" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 41909 } } }
I20260811 11:20:23.539640 20547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:23.539650 20535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:23.540215 20547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:23.541960 20547 catalog_manager.cc:1347] Generated new cluster ID: 8375646526c54a988c269704fe35b76b
I20260811 11:20:23.542011 20547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 11:20:23 dist-test-slave-2w3w krb5kdc[20437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:23.551265 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:23.538901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35127 (local address 127.28.25.62:41909)
0811 11:20:23.539134 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:23.539144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:23.539243 (+ 99us) server_negotiation.cc:407] Connection header received
0811 11:20:23.539467 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:23.539478 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:23.539540 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:23.539679 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:23.540869 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:23.542171 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:23.543264 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:23.543898 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:23.544006 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:23.546956 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:23.546987 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:23.546988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:23.547004 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:23.549256 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:23.549767 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:23.549771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:23.549772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:23.549940 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:23.550187 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:23.550188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:23.550188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:23.550576 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:23.550760 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:23.550851 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 11:20:23.551001 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":111,"threads_started":1}
I20260811 11:20:23.552356 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:23.561690 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:23.553944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59555 (local address 127.28.25.62:41909)
0811 11:20:23.553991 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:23.553993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:23.554160 (+ 167us) server_negotiation.cc:407] Connection header received
0811 11:20:23.554191 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:23.554192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:23.554204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:23.554276 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:23.554956 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:23.555824 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:23.556837 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:23.557172 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:23.557256 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:23.558700 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:23.558702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:23.558702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:23.558710 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:23.560397 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:23.560831 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:23.560833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:23.560833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:23.560899 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:23.561139 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:23.561140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:23.561140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:23.561295 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:23.561404 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:23.561523 (+ 119us) server_negotiation.cc:299] Negotiation successful
0811 11:20:23.561589 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 11:20:23.563302 20547 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:23.564005 20547 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:23.572192 20547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21e8c0c723c8465cbf91446f88026c61: Generated new TSK 0
I20260811 11:20:23.572489 20547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:23.582715 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 20451
W20260811 11:20:23.591881 20435 connection.cc:535] client connection to 127.28.25.62:41909 recv error: Network error: failed to read from TLS socket (remote: 127.28.25.62:41909): Connection reset by peer (error 104)
2026-08-11T11:20:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1523 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:23 dist-test-slave-2w3w krb5kdc[20565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:23 dist-test-slave-2w3w krb5kdc[20565](info): set up 2 sockets
Aug 11 11:20:23 dist-test-slave-2w3w krb5kdc[20565](info): commencing operation
krb5kdc: starting...
W20260811 11:20:24.755895 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:24 dist-test-slave-2w3w krb5kdc[20565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:20:24.899597 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39331
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:43063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:39331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:24.987939 20583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:24.988198 20583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:24.988236 20583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:24.989573 20583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:24.989640 20583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:24.989656 20583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:24.989670 20583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:24.989682 20583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:24.991549 20583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:43063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:39331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:24.991888 20583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:24.992183 20583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:24.998047 20588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:24.998147 20592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:24.998055 20583 server_base.cc:1034] running on GCE node
W20260811 11:20:24.998070 20590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:24.998566 20583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:24.999022 20583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:25.000240 20583 hybrid_clock.cc:648] HybridClock initialized: now 1786447225000232 us; error 38 us; skew 500 ppm
Aug 11 11:20:24 dist-test-slave-2w3w krb5kdc[20565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447224, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:25.003367 20583 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:20:25.003942 20583 webserver.cc:460] Webserver started at http://127.28.25.62:44011/ using document root <none> and password file <none>
I20260811 11:20:25.004217 20583 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:25.004297 20583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:25.004478 20583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:25.005815 20583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "2a8f81e300314c1a9c4ce806e09e82d5"
format_stamp: "Formatted at 2026-08-11 11:20:25 on dist-test-slave-2w3w"
server_key: "61176b2fc35fbe197630eaceddac71cb"
server_key_iv: "59610021c8231d0dcabea1a8fa1879ca"
server_key_version: "encryptionkey@0"
I20260811 11:20:25.006275 20583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "2a8f81e300314c1a9c4ce806e09e82d5"
format_stamp: "Formatted at 2026-08-11 11:20:25 on dist-test-slave-2w3w"
server_key: "61176b2fc35fbe197630eaceddac71cb"
server_key_iv: "59610021c8231d0dcabea1a8fa1879ca"
server_key_version: "encryptionkey@0"
I20260811 11:20:25.008085 20583 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 11:20:25.009135 20599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:25.009390 20583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:25.009531 20583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "2a8f81e300314c1a9c4ce806e09e82d5"
format_stamp: "Formatted at 2026-08-11 11:20:25 on dist-test-slave-2w3w"
server_key: "61176b2fc35fbe197630eaceddac71cb"
server_key_iv: "59610021c8231d0dcabea1a8fa1879ca"
server_key_version: "encryptionkey@0"
I20260811 11:20:25.009631 20583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:25.026497 20583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:25.029070 20583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:25.029254 20583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:25.033859 20583 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:39331
I20260811 11:20:25.033896 20658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:39331 every 8 connection(s)
I20260811 11:20:25.034294 20583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:25.035440 20659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:25.036181 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 20583
I20260811 11:20:25.036334 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:25.036530 28772 external_mini_cluster.cc:1428] Setting key 4b3d4105e97594335c1ac0e4f7865be1
I20260811 11:20:25.038775 20659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5: Bootstrap starting.
I20260811 11:20:25.039621 20659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:25.039979 20659 log.cc:824] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:25.040846 20659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5: No bootstrap required, opened a new log
I20260811 11:20:25.042658 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:25.037128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47177 (local address 127.28.25.62:39331)
0811 11:20:25.037344 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:25.037352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:25.037380 (+ 28us) server_negotiation.cc:407] Connection header received
0811 11:20:25.037870 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:25.037877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:25.037929 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:25.038049 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:25.038324 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:25.039357 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:25.040458 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:25.040950 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:25.041061 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:25.041884 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:25.041888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:25.041888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:25.042063 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:25.042101 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:25.042113 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:25.042199 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:25.042335 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 11:20:25.042395 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
I20260811 11:20:25.043519 20659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8f81e300314c1a9c4ce806e09e82d5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39331 } }
I20260811 11:20:25.043715 20659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:25.043768 20659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8f81e300314c1a9c4ce806e09e82d5, State: Initialized, Role: FOLLOWER
I20260811 11:20:25.043910 20659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8f81e300314c1a9c4ce806e09e82d5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39331 } }
I20260811 11:20:25.044025 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:25.044047 20659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:25.044085 20659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:25.044135 20659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:25.045109 20659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8f81e300314c1a9c4ce806e09e82d5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39331 } }
I20260811 11:20:25.045332 20659 leader_election.cc:302] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a8f81e300314c1a9c4ce806e09e82d5; no voters:
I20260811 11:20:25.045578 20659 leader_election.cc:288] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:25.045615 20669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:25.045881 20669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [term 1 LEADER]: Becoming Leader. State: Replica: 2a8f81e300314c1a9c4ce806e09e82d5, State: Running, Role: LEADER
I20260811 11:20:25.045982 20659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:25.046002 20669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8f81e300314c1a9c4ce806e09e82d5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39331 } }
I20260811 11:20:25.046389 20669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a8f81e300314c1a9c4ce806e09e82d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8f81e300314c1a9c4ce806e09e82d5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39331 } } }
I20260811 11:20:25.046506 20669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:25.046672 20669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a8f81e300314c1a9c4ce806e09e82d5. Latest consensus state: current_term: 1 leader_uuid: "2a8f81e300314c1a9c4ce806e09e82d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8f81e300314c1a9c4ce806e09e82d5" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39331 } } }
I20260811 11:20:25.046772 20669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5 [sys.catalog]: This master's current role is: LEADER
W20260811 11:20:25.049305 20687 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:25.049436 20687 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:25.049546 20677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:25.049733 20677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:25.051218 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:25.045771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40971 (local address 127.28.25.62:39331)
0811 11:20:25.045930 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:25.045933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:25.046328 (+ 395us) server_negotiation.cc:407] Connection header received
0811 11:20:25.046477 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:25.046478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:25.046517 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:25.046551 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:25.046907 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:25.047764 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:25.049101 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:25.049493 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:25.049572 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:25.050854 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:25.050856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:25.050856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:25.050862 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:25.050881 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:25.050886 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:25.050979 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:25.051103 (+ 124us) server_negotiation.cc:299] Negotiation successful
0811 11:20:25.051117 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260811 11:20:25.052733 20677 catalog_manager.cc:1347] Generated new cluster ID: 0628906bac7840fabf5154c2ca39ef78
I20260811 11:20:25.052790 20677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:25.078594 20677 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:25.079334 20677 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:25.088650 20677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a8f81e300314c1a9c4ce806e09e82d5: Generated new TSK 0
I20260811 11:20:25.088919 20677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:25.098524 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 20583
W20260811 11:20:25.108500 20563 connection.cc:535] client connection to 127.28.25.62:39331 recv error: Network error: failed to read from TLS socket (remote: 127.28.25.62:39331): Connection reset by peer (error 104)
2026-08-11T11:20:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1516 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:25 dist-test-slave-2w3w krb5kdc[20694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:25 dist-test-slave-2w3w krb5kdc[20694](info): set up 2 sockets
Aug 11 11:20:25 dist-test-slave-2w3w krb5kdc[20694](info): commencing operation
krb5kdc: starting...
W20260811 11:20:26.232163 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:26 dist-test-slave-2w3w krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:20:26.373747 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:46099
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:46099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:26.458683 20708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:26.458983 20708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:26.459077 20708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:26.460413 20708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:26.460505 20708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:26.460529 20708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:26.460551 20708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:26.460579 20708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:26.462271 20708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:46099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:46099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:26.462615 20708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:26.462878 20708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:26.468147 20715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:26.468151 20713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:26.468532 20708 server_base.cc:1034] running on GCE node
W20260811 11:20:26.468205 20717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:26.468856 20708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:26.469199 20708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:26.470405 20708 hybrid_clock.cc:648] HybridClock initialized: now 1786447226470432 us; error 74 us; skew 500 ppm
Aug 11 11:20:26 dist-test-slave-2w3w krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447226, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:26.473593 20708 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:20:26.474162 20708 webserver.cc:460] Webserver started at http://127.28.25.62:42485/ using document root <none> and password file <none>
I20260811 11:20:26.474462 20708 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:26.474540 20708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:26.474702 20708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:26.476003 20708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "9c29f8ebe4414c89baf1522cfc4a8483"
format_stamp: "Formatted at 2026-08-11 11:20:26 on dist-test-slave-2w3w"
server_key: "4890b9b39779f3871a3c58feb65cc88c"
server_key_iv: "25388cee817480c30df9b7729387ebfa"
server_key_version: "encryptionkey@0"
I20260811 11:20:26.476404 20708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "9c29f8ebe4414c89baf1522cfc4a8483"
format_stamp: "Formatted at 2026-08-11 11:20:26 on dist-test-slave-2w3w"
server_key: "4890b9b39779f3871a3c58feb65cc88c"
server_key_iv: "25388cee817480c30df9b7729387ebfa"
server_key_version: "encryptionkey@0"
I20260811 11:20:26.477931 20708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:20:26.478940 20726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:26.479205 20708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:26.479342 20708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "9c29f8ebe4414c89baf1522cfc4a8483"
format_stamp: "Formatted at 2026-08-11 11:20:26 on dist-test-slave-2w3w"
server_key: "4890b9b39779f3871a3c58feb65cc88c"
server_key_iv: "25388cee817480c30df9b7729387ebfa"
server_key_version: "encryptionkey@0"
I20260811 11:20:26.479470 20708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:26.483578 20708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:26.486411 20708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:26.486603 20708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:26.491683 20708 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:46099
I20260811 11:20:26.491747 20784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:46099 every 8 connection(s)
I20260811 11:20:26.492165 20708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:26.493376 20787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:26.496217 20787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483: Bootstrap starting.
I20260811 11:20:26.496977 20787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:26.497346 20787 log.cc:824] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:26.498266 20787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483: No bootstrap required, opened a new log
I20260811 11:20:26.499998 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 20708
I20260811 11:20:26.500155 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:26.500371 28772 external_mini_cluster.cc:1428] Setting key 62ba9399bd53d9ad301672d49c76e2a6
I20260811 11:20:26.500463 20787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c29f8ebe4414c89baf1522cfc4a8483" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46099 } }
I20260811 11:20:26.500675 20787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:26.500731 20787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c29f8ebe4414c89baf1522cfc4a8483, State: Initialized, Role: FOLLOWER
I20260811 11:20:26.500942 20787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c29f8ebe4414c89baf1522cfc4a8483" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46099 } }
I20260811 11:20:26.501045 20787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:26.501073 20787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:26.501150 20787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:26.502558 20787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c29f8ebe4414c89baf1522cfc4a8483" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46099 } }
I20260811 11:20:26.502697 20787 leader_election.cc:302] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c29f8ebe4414c89baf1522cfc4a8483; no voters:
I20260811 11:20:26.502950 20787 leader_election.cc:288] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:26.503041 20792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:26.503348 20787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:26.503324 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:26.501011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36715 (local address 127.28.25.62:46099)
0811 11:20:26.501273 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:26.501284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:26.501336 (+ 52us) server_negotiation.cc:407] Connection header received
0811 11:20:26.501515 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:26.501523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:26.501558 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:26.501757 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:26.502561 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:26.502565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:26.502565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:26.502571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:26.502629 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:26.502648 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:26.502737 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:26.502949 (+ 212us) server_negotiation.cc:299] Negotiation successful
0811 11:20:26.503043 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":110,"threads_started":1}
I20260811 11:20:26.503298 20792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [term 1 LEADER]: Becoming Leader. State: Replica: 9c29f8ebe4414c89baf1522cfc4a8483, State: Running, Role: LEADER
I20260811 11:20:26.503620 20792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c29f8ebe4414c89baf1522cfc4a8483" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46099 } }
I20260811 11:20:26.504086 20793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c29f8ebe4414c89baf1522cfc4a8483. Latest consensus state: current_term: 1 leader_uuid: "9c29f8ebe4414c89baf1522cfc4a8483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c29f8ebe4414c89baf1522cfc4a8483" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46099 } } }
I20260811 11:20:26.504086 20794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c29f8ebe4414c89baf1522cfc4a8483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c29f8ebe4414c89baf1522cfc4a8483" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 46099 } } }
I20260811 11:20:26.504267 20793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:26.504276 20794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:26.504565 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:26.505100 20801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:26.505753 20801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:26.507747 20801 catalog_manager.cc:1347] Generated new cluster ID: caad513281274154bae4b2162b168795
I20260811 11:20:26.507794 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:26.506379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44045 (local address 127.28.25.62:46099)
0811 11:20:26.506426 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:26.506428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:26.506575 (+ 147us) server_negotiation.cc:407] Connection header received
0811 11:20:26.506664 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:26.506665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:26.506688 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:26.506742 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:26.507475 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:26.507476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:26.507476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:26.507482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:26.507498 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:26.507502 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:26.507582 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:26.507633 (+ 51us) server_negotiation.cc:299] Negotiation successful
0811 11:20:26.507649 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 11:20:26.507798 20801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:26.550319 20801 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:26.550565 20801 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:26.556340 20801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c29f8ebe4414c89baf1522cfc4a8483: Generated new TSK 0
I20260811 11:20:26.556643 20801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:26.562177 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 20708
2026-08-11T11:20:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:26 dist-test-slave-2w3w krb5kdc[20819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:26 dist-test-slave-2w3w krb5kdc[20819](info): set up 2 sockets
Aug 11 11:20:26 dist-test-slave-2w3w krb5kdc[20819](info): commencing operation
krb5kdc: starting...
W20260811 11:20:27.691753 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:27 dist-test-slave-2w3w krb5kdc[20819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:20:27.834535 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:44185
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:44185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:27.931667 20832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:27.931916 20832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:27.931944 20832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:27.933146 20832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:27.933194 20832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:27.933209 20832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:27.933221 20832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:27.933236 20832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:27.934803 20832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:44185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:44185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:27.935122 20832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:27.935338 20832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:27.940622 20840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:27.940694 20839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:27.940600 20842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:27.940752 20832 server_base.cc:1034] running on GCE node
I20260811 11:20:27.941190 20832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:27.941555 20832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:27.942755 20832 hybrid_clock.cc:648] HybridClock initialized: now 1786447227942751 us; error 47 us; skew 500 ppm
Aug 11 11:20:27 dist-test-slave-2w3w krb5kdc[20819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:27.946098 20832 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:20:27.946671 20832 webserver.cc:460] Webserver started at http://127.28.25.62:39993/ using document root <none> and password file <none>
I20260811 11:20:27.947034 20832 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:27.947121 20832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:27.947311 20832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:27.948599 20832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "944b5750611c4f5292a0fe31eb81b9db"
format_stamp: "Formatted at 2026-08-11 11:20:27 on dist-test-slave-2w3w"
server_key: "cc57b624c9be90ffe348591be14fe7f7"
server_key_iv: "47db2c62845bd36ecd5682f7a648d55f"
server_key_version: "encryptionkey@0"
I20260811 11:20:27.949044 20832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "944b5750611c4f5292a0fe31eb81b9db"
format_stamp: "Formatted at 2026-08-11 11:20:27 on dist-test-slave-2w3w"
server_key: "cc57b624c9be90ffe348591be14fe7f7"
server_key_iv: "47db2c62845bd36ecd5682f7a648d55f"
server_key_version: "encryptionkey@0"
I20260811 11:20:27.950703 20832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:20:27.951743 20850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:27.952009 20832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:27.952104 20832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "944b5750611c4f5292a0fe31eb81b9db"
format_stamp: "Formatted at 2026-08-11 11:20:27 on dist-test-slave-2w3w"
server_key: "cc57b624c9be90ffe348591be14fe7f7"
server_key_iv: "47db2c62845bd36ecd5682f7a648d55f"
server_key_version: "encryptionkey@0"
I20260811 11:20:27.952164 20832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:27.967707 20832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:27.970196 20832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:27.970340 20832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:27.974915 20832 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:44185
I20260811 11:20:27.975062 20911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:44185 every 8 connection(s)
I20260811 11:20:27.975419 20832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:27.976447 20912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:27.978885 20912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db: Bootstrap starting.
I20260811 11:20:27.979557 20912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:27.979832 20912 log.cc:824] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:27.980608 20912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db: No bootstrap required, opened a new log
I20260811 11:20:27.981343 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 20832
I20260811 11:20:27.981469 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:27.981664 28772 external_mini_cluster.cc:1428] Setting key e67d9c0ee394bad5c9627331cb65cddd
I20260811 11:20:27.983637 20912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944b5750611c4f5292a0fe31eb81b9db" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44185 } }
I20260811 11:20:27.983887 20912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:27.983937 20912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944b5750611c4f5292a0fe31eb81b9db, State: Initialized, Role: FOLLOWER
I20260811 11:20:27.984103 20912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944b5750611c4f5292a0fe31eb81b9db" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44185 } }
I20260811 11:20:27.984231 20912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:27.984287 20912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:27.984357 20912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:27.985321 20912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944b5750611c4f5292a0fe31eb81b9db" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44185 } }
I20260811 11:20:27.985471 20912 leader_election.cc:302] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 944b5750611c4f5292a0fe31eb81b9db; no voters:
I20260811 11:20:27.985724 20912 leader_election.cc:288] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:27.985761 20918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:27.985895 20918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [term 1 LEADER]: Becoming Leader. State: Replica: 944b5750611c4f5292a0fe31eb81b9db, State: Running, Role: LEADER
I20260811 11:20:27.986066 20912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:27.986063 20918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944b5750611c4f5292a0fe31eb81b9db" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44185 } }
I20260811 11:20:27.987453 20919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "944b5750611c4f5292a0fe31eb81b9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944b5750611c4f5292a0fe31eb81b9db" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44185 } } }
I20260811 11:20:27.987571 20919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:27.987787 20920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 944b5750611c4f5292a0fe31eb81b9db. Latest consensus state: current_term: 1 leader_uuid: "944b5750611c4f5292a0fe31eb81b9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944b5750611c4f5292a0fe31eb81b9db" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 44185 } } }
I20260811 11:20:27.988168 20920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db [sys.catalog]: This master's current role is: LEADER
W20260811 11:20:27.988245 20932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:27.988318 20932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:27.988430 20934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:27.988723 20934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 11 11:20:27 dist-test-slave-2w3w krb5kdc[20819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:27.990707 20934 catalog_manager.cc:1347] Generated new cluster ID: 0005afc111c54f0a8f96c450686c55da
I20260811 11:20:27.990769 20934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:27.995604 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:27.982284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56975 (local address 127.28.25.62:44185)
0811 11:20:27.982482 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:27.982491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:27.982520 (+ 29us) server_negotiation.cc:407] Connection header received
0811 11:20:27.983554 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:27.983564 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:27.983624 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:27.983754 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:27.984764 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:27.986080 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:27.987324 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:27.988023 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:27.990013 (+ 1990us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:27.991260 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:27.991265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:27.991265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:27.991278 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:27.993559 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:27.994061 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:27.994066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:27.994066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:27.994204 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:27.994456 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:27.994457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:27.994457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:27.994827 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:27.995043 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:27.995171 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 11:20:27.995349 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260811 11:20:27.995783 20934 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:27.996773 20934 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:27.997148 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:28.003520 20934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 944b5750611c4f5292a0fe31eb81b9db: Generated new TSK 0
I20260811 11:20:28.003736 20934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:28.006685 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:27.998749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36677 (local address 127.28.25.62:44185)
0811 11:20:27.998792 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:27.998794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:27.999007 (+ 213us) server_negotiation.cc:407] Connection header received
0811 11:20:27.999066 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:27.999067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:27.999079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:27.999143 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:27.999816 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:28.000623 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:28.001670 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:28.002026 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:28.002105 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:28.003717 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:28.003718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:28.003718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:28.003725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:28.005455 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:28.005803 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:28.005804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:28.005804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:28.005871 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:28.006057 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:28.006058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:28.006058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:28.006192 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:28.006291 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:28.006481 (+ 190us) server_negotiation.cc:299] Negotiation successful
0811 11:20:28.006573 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:20:28.009090 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 20832
2026-08-11T11:20:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:28 dist-test-slave-2w3w krb5kdc[20947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:28 dist-test-slave-2w3w krb5kdc[20947](info): set up 2 sockets
Aug 11 11:20:28 dist-test-slave-2w3w krb5kdc[20947](info): commencing operation
krb5kdc: starting...
W20260811 11:20:29.166066 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:29 dist-test-slave-2w3w krb5kdc[20947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:20:29.306571 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:36493
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:36493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:29.392216 20960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:29.392450 20960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:29.392478 20960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:29.393612 20960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:29.393657 20960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:29.393680 20960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:29.393700 20960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:29.393712 20960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:29.395234 20960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:36493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:36493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:29.395448 20960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:29.395643 20960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:29.400900 20966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:29.400900 20969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:29.400943 20967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:29.401297 20960 server_base.cc:1034] running on GCE node
I20260811 11:20:29.401595 20960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:29.401876 20960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:29.403072 20960 hybrid_clock.cc:648] HybridClock initialized: now 1786447229403047 us; error 36 us; skew 500 ppm
Aug 11 11:20:29 dist-test-slave-2w3w krb5kdc[20947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447229, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:29.409932 20960 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:20:29.410537 20960 webserver.cc:460] Webserver started at http://127.28.25.62:42795/ using document root <none> and password file <none>
I20260811 11:20:29.410806 20960 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:29.410867 20960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:29.411124 20960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:29.412367 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "8de410cae0074684a006cce34d57ce65"
format_stamp: "Formatted at 2026-08-11 11:20:29 on dist-test-slave-2w3w"
server_key: "a882e66bc4f9d74e2e4ff4f020795a97"
server_key_iv: "99fa79240ba365eb7857dc01e7385916"
server_key_version: "encryptionkey@0"
I20260811 11:20:29.412765 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "8de410cae0074684a006cce34d57ce65"
format_stamp: "Formatted at 2026-08-11 11:20:29 on dist-test-slave-2w3w"
server_key: "a882e66bc4f9d74e2e4ff4f020795a97"
server_key_iv: "99fa79240ba365eb7857dc01e7385916"
server_key_version: "encryptionkey@0"
I20260811 11:20:29.414251 20960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 11:20:29.415200 20976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:29.415439 20960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:29.415545 20960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "8de410cae0074684a006cce34d57ce65"
format_stamp: "Formatted at 2026-08-11 11:20:29 on dist-test-slave-2w3w"
server_key: "a882e66bc4f9d74e2e4ff4f020795a97"
server_key_iv: "99fa79240ba365eb7857dc01e7385916"
server_key_version: "encryptionkey@0"
I20260811 11:20:29.415629 20960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:29.448050 20960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:29.451254 20960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:29.451442 20960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:29.456967 20960 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:36493
I20260811 11:20:29.456997 21034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:36493 every 8 connection(s)
I20260811 11:20:29.457579 20960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:29.458842 21035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:29.462709 21035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65: Bootstrap starting.
I20260811 11:20:29.463686 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 20960
I20260811 11:20:29.463742 21035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:29.463829 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:29.464077 28772 external_mini_cluster.cc:1428] Setting key 82a8cc41eed3fd640465deda0a5370bd
I20260811 11:20:29.464114 21035 log.cc:824] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:29.465047 21035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65: No bootstrap required, opened a new log
I20260811 11:20:29.468133 21035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de410cae0074684a006cce34d57ce65" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36493 } }
I20260811 11:20:29.468365 21035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:29.468434 21035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de410cae0074684a006cce34d57ce65, State: Initialized, Role: FOLLOWER
I20260811 11:20:29.468585 21035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de410cae0074684a006cce34d57ce65" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36493 } }
I20260811 11:20:29.468674 21035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:29.468735 21035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:29.468792 21035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:29.469802 21035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de410cae0074684a006cce34d57ce65" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36493 } }
I20260811 11:20:29.469986 21035 leader_election.cc:302] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8de410cae0074684a006cce34d57ce65; no voters:
I20260811 11:20:29.470211 21035 leader_election.cc:288] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:29.470377 21042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:29.470506 21035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:29.470724 21042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [term 1 LEADER]: Becoming Leader. State: Replica: 8de410cae0074684a006cce34d57ce65, State: Running, Role: LEADER
I20260811 11:20:29.470873 21042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de410cae0074684a006cce34d57ce65" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36493 } }
I20260811 11:20:29.471349 21043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8de410cae0074684a006cce34d57ce65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de410cae0074684a006cce34d57ce65" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36493 } } }
I20260811 11:20:29.471339 21042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8de410cae0074684a006cce34d57ce65. Latest consensus state: current_term: 1 leader_uuid: "8de410cae0074684a006cce34d57ce65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de410cae0074684a006cce34d57ce65" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 36493 } } }
I20260811 11:20:29.471465 21043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:29.471500 21042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:29.471782 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:29.464784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44587 (local address 127.28.25.62:36493)
0811 11:20:29.464954 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:29.464964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:29.464995 (+ 31us) server_negotiation.cc:407] Connection header received
0811 11:20:29.465536 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:29.465544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:29.465597 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:29.465760 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:29.466041 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:29.467502 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:29.468711 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:29.469336 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:29.469876 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:29.470433 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:29.470438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:29.470438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:29.470687 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:29.470738 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:29.470758 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:29.471127 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:29.471160 (+ 33us) server_negotiation.cc:299] Negotiation successful
0811 11:20:29.471240 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
W20260811 11:20:29.472584 21056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:29.472638 21056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:29.472743 21050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:29.472980 21050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:29.472988 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:29.474571 21050 catalog_manager.cc:1347] Generated new cluster ID: b61b8710e0dd4ba78ee84559be07f4e8
I20260811 11:20:29.474625 21050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:29.491489 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:29.479280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59567 (local address 127.28.25.62:36493)
0811 11:20:29.479337 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:29.479339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:29.487389 (+ 8050us) server_negotiation.cc:407] Connection header received
0811 11:20:29.487474 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:29.487476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:29.487491 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:29.487547 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:29.487733 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:29.488658 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:29.489925 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:29.490260 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:29.490338 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:29.491170 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:29.491172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:29.491172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:29.491178 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:29.491196 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:29.491201 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:29.491252 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:29.491327 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 11:20:29.491354 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 11:20:29.509416 21050 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:29.510186 21050 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:29.512611 21050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8de410cae0074684a006cce34d57ce65: Generated new TSK 0
I20260811 11:20:29.512805 21050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:29.522109 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 20960
2026-08-11T11:20:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1514 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:29 dist-test-slave-2w3w krb5kdc[21067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:29 dist-test-slave-2w3w krb5kdc[21067](info): set up 2 sockets
Aug 11 11:20:29 dist-test-slave-2w3w krb5kdc[21067](info): commencing operation
krb5kdc: starting...
W20260811 11:20:30.637923 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:30 dist-test-slave-2w3w krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:20:30.780176 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39025
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:30.866464 21081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:30.866740 21081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:30.866788 21081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:30.868188 21081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:30.868287 21081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:30.868312 21081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:30.868335 21081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:30.868373 21081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:30.870105 21081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:39025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:30.870455 21081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:30.870723 21081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:30.876334 21090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:30.876439 21081 server_base.cc:1034] running on GCE node
W20260811 11:20:30.876377 21088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:30.876334 21087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:30.876981 21081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:30.877346 21081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:30.878543 21081 hybrid_clock.cc:648] HybridClock initialized: now 1786447230878521 us; error 42 us; skew 500 ppm
Aug 11 11:20:30 dist-test-slave-2w3w krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:30.881489 21081 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:20:30.881935 21081 webserver.cc:460] Webserver started at http://127.28.25.62:36057/ using document root <none> and password file <none>
I20260811 11:20:30.882133 21081 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:30.882174 21081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:30.882272 21081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:30.883488 21081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "f738c29dd697409692e312fc611541f0"
format_stamp: "Formatted at 2026-08-11 11:20:30 on dist-test-slave-2w3w"
server_key: "5f001f1435fb674dd5af397fbc54f748"
server_key_iv: "fc1cb2134510033985f173ad487b8575"
server_key_version: "encryptionkey@0"
I20260811 11:20:30.883826 21081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "f738c29dd697409692e312fc611541f0"
format_stamp: "Formatted at 2026-08-11 11:20:30 on dist-test-slave-2w3w"
server_key: "5f001f1435fb674dd5af397fbc54f748"
server_key_iv: "fc1cb2134510033985f173ad487b8575"
server_key_version: "encryptionkey@0"
I20260811 11:20:30.885380 21081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:20:30.886240 21097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:30.886499 21081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:30.886593 21081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "f738c29dd697409692e312fc611541f0"
format_stamp: "Formatted at 2026-08-11 11:20:30 on dist-test-slave-2w3w"
server_key: "5f001f1435fb674dd5af397fbc54f748"
server_key_iv: "fc1cb2134510033985f173ad487b8575"
server_key_version: "encryptionkey@0"
I20260811 11:20:30.886660 21081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:30.890409 21081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:30.892856 21081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:30.892984 21081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:30.897956 21081 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:39025
I20260811 11:20:30.897991 21154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:39025 every 8 connection(s)
I20260811 11:20:30.898386 21081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:30.899722 21155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:30.902237 21155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0: Bootstrap starting.
I20260811 11:20:30.902916 21155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:30.903266 21155 log.cc:824] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:30.904142 21155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0: No bootstrap required, opened a new log
I20260811 11:20:30.905819 21155 raft_consensus.cc:348] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f738c29dd697409692e312fc611541f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39025 } }
I20260811 11:20:30.905997 21155 raft_consensus.cc:374] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:20:30.906023 21155 raft_consensus.cc:729] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738c29dd697409692e312fc611541f0, State: Initialized, Role: FOLLOWER
I20260811 11:20:30.906137 21155 consensus_queue.cc:260] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f738c29dd697409692e312fc611541f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39025 } }
I20260811 11:20:30.906315 21155 raft_consensus.cc:388] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:20:30.906356 21155 raft_consensus.cc:482] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:20:30.906418 21155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:30.906674 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 21081
I20260811 11:20:30.906765 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:30.907011 28772 external_mini_cluster.cc:1428] Setting key 752a353e1fd14d67ff851355967edd62
I20260811 11:20:30.907920 21155 raft_consensus.cc:504] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f738c29dd697409692e312fc611541f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39025 } }
I20260811 11:20:30.908049 21155 leader_election.cc:302] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f738c29dd697409692e312fc611541f0; no voters:
I20260811 11:20:30.908205 21155 leader_election.cc:288] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:20:30.908298 21160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:30.908501 21155 sys_catalog.cc:564] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:30.908514 21160 raft_consensus.cc:686] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [term 1 LEADER]: Becoming Leader. State: Replica: f738c29dd697409692e312fc611541f0, State: Running, Role: LEADER
I20260811 11:20:30.909055 21160 consensus_queue.cc:237] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f738c29dd697409692e312fc611541f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39025 } }
I20260811 11:20:30.909687 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:30.907689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48593 (local address 127.28.25.62:39025)
0811 11:20:30.907912 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:30.907923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:30.907969 (+ 46us) server_negotiation.cc:407] Connection header received
0811 11:20:30.908103 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:30.908111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:30.908146 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:30.908312 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:30.909150 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:30.909154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:30.909155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:30.909161 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:30.909207 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:30.909222 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:30.909289 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:30.909377 (+ 88us) server_negotiation.cc:299] Negotiation successful
0811 11:20:30.909429 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260811 11:20:30.909988 21161 sys_catalog.cc:455] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f738c29dd697409692e312fc611541f0. Latest consensus state: current_term: 1 leader_uuid: "f738c29dd697409692e312fc611541f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f738c29dd697409692e312fc611541f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39025 } } }
I20260811 11:20:30.910194 21161 sys_catalog.cc:458] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:30.909531 21162 sys_catalog.cc:455] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f738c29dd697409692e312fc611541f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f738c29dd697409692e312fc611541f0" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 39025 } } }
I20260811 11:20:30.910305 21162 sys_catalog.cc:458] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:30.910840 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260811 11:20:30.911000 21176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:30.911051 21176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:30.911103 21177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:30.911293 21177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:30.913137 21177 catalog_manager.cc:1347] Generated new cluster ID: 2ac40ad7f26245848ff734e4dfc8c2df
I20260811 11:20:30.913205 21177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:30.913909 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:30.912599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35665 (local address 127.28.25.62:39025)
0811 11:20:30.912640 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:30.912642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:30.912902 (+ 260us) server_negotiation.cc:407] Connection header received
0811 11:20:30.912946 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:30.912946 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:30.912962 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:30.913015 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:30.913690 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:30.913691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:30.913691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:20:30.913695 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:30.913710 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:20:30.913715 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:30.913758 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:30.913817 (+ 59us) server_negotiation.cc:299] Negotiation successful
0811 11:20:30.913830 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 11:20:30.936764 21177 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:30.937006 21177 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:30.945487 21177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f738c29dd697409692e312fc611541f0: Generated new TSK 0
I20260811 11:20:30.945772 21177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:30.956403 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 21081
2026-08-11T11:20:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1433 ms)
[----------] 18 tests from AuthTokenIssuingTest (26244 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:31 dist-test-slave-2w3w krb5kdc[21188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:31 dist-test-slave-2w3w krb5kdc[21188](info): set up 2 sockets
Aug 11 11:20:31 dist-test-slave-2w3w krb5kdc[21188](info): commencing operation
krb5kdc: starting...
W20260811 11:20:32.078814 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:20:32.223239 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:38277
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:38277,127.28.25.61:41591,127.28.25.60:46177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:32.310715 21202 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:20:32.311048 21202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:32.311110 21202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:32.311156 21202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:32.312469 21202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:32.312558 21202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:32.312582 21202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:32.312605 21202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:32.312637 21202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:32.314311 21202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:38277,127.28.25.61:41591,127.28.25.60:46177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:32.314630 21202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:32.314910 21202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:32.320636 21210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:32.320645 21208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:32.320645 21207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:32.320708 21202 server_base.cc:1034] running on GCE node
I20260811 11:20:32.321133 21202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:32.321432 21202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:32.322589 21202 hybrid_clock.cc:648] HybridClock initialized: now 1786447232322545 us; error 83 us; skew 500 ppm
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:32.325732 21202 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:20:32.326335 21202 webserver.cc:460] Webserver started at http://127.28.25.62:42741/ using document root <none> and password file <none>
I20260811 11:20:32.326589 21202 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:32.326661 21202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:32.326830 21202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:32.328050 21202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "8906afc902b9413e9ea54fec80415b3e"
format_stamp: "Formatted at 2026-08-11 11:20:32 on dist-test-slave-2w3w"
server_key: "6cb9cbfef93cec6d0af0e1a0cff5f31f"
server_key_iv: "fae0aff1bb187936bccaf419c8d51259"
server_key_version: "encryptionkey@0"
I20260811 11:20:32.328435 21202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "8906afc902b9413e9ea54fec80415b3e"
format_stamp: "Formatted at 2026-08-11 11:20:32 on dist-test-slave-2w3w"
server_key: "6cb9cbfef93cec6d0af0e1a0cff5f31f"
server_key_iv: "fae0aff1bb187936bccaf419c8d51259"
server_key_version: "encryptionkey@0"
I20260811 11:20:32.329900 21202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:20:32.330866 21217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:32.331125 21202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:32.331259 21202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "8906afc902b9413e9ea54fec80415b3e"
format_stamp: "Formatted at 2026-08-11 11:20:32 on dist-test-slave-2w3w"
server_key: "6cb9cbfef93cec6d0af0e1a0cff5f31f"
server_key_iv: "fae0aff1bb187936bccaf419c8d51259"
server_key_version: "encryptionkey@0"
I20260811 11:20:32.331352 21202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:32.340723 21202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:32.343426 21202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:32.343640 21202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:32.349458 21202 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:38277
I20260811 11:20:32.349479 21276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:38277 every 8 connection(s)
I20260811 11:20:32.349982 21202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:32.351143 21277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:32.352953 21277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:32.360036 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 21202
I20260811 11:20:32.360183 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:32.360407 28772 external_mini_cluster.cc:1428] Setting key 4693e1d4d316c64720dacb8ae5dfd935
I20260811 11:20:32.363695 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.353728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:38277 (local address 127.0.0.1:40854)
0811 11:20:32.354005 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.354053 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:32.354599 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:32.354995 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:32.354997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:32.355326 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:32.355493 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.355498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.356547 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.356548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.357255 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.357257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.357793 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.357794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.357867 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.358448 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:32.358457 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:32.360041 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:32.362056 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.362062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.362065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.362270 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.362530 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.362531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.362531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.362626 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.363059 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:32.363066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:32.363136 (+ 70us) client_negotiation.cc:769] Sending connection context
0811 11:20:32.363250 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 11:20:32.363502 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":93,"thread_start_us":164,"threads_started":1}
I20260811 11:20:32.363703 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.353766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.28.25.62:38277)
0811 11:20:32.354016 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.354025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.354166 (+ 141us) server_negotiation.cc:407] Connection header received
0811 11:20:32.354786 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.354788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.354806 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.354991 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.355608 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.356444 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.357347 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.357667 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.357806 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.360140 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.360148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.360148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.360156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.361957 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.362366 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.362369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.362369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.362461 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.362695 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.362696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.362696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.362947 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.363084 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.363284 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.363419 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
I20260811 11:20:32.364739 21277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:20:32.365195 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.365065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41591 (local address 127.0.0.1:57276)
0811 11:20:32.365117 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.365163 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260811 11:20:32.365327 21220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.25.61:41591: connect: Connection refused (error 111)
I20260811 11:20:32.365792 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.365692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41591 (local address 127.0.0.1:57288)
0811 11:20:32.365744 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.365768 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:20:32.365903 21277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.61:41591: Network error: Client connection negotiation failed: client connection to 127.28.25.61:41591: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.28.25.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab.
Entry for principal kudu/127.28.25.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab.
I20260811 11:20:32.391559 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.61:41591
--webserver_interface=127.28.25.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab
--principal=kudu/127.28.25.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:38277,127.28.25.61:41591,127.28.25.60:46177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:20:32.415107 21277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } attempt: 1
I20260811 11:20:32.415637 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.415491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41591 (local address 127.0.0.1:57298)
0811 11:20:32.415563 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.415606 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260811 11:20:32.416071 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.415964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41591 (local address 127.0.0.1:57308)
0811 11:20:32.416016 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.416045 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:20:32.416174 21277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.61:41591: Network error: Client connection negotiation failed: client connection to 127.28.25.61:41591: connect: Connection refused (error 111)
W20260811 11:20:32.478708 21286 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:20:32.479084 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:32.479126 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:32.479176 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:32.480536 21286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:32.480629 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:32.480654 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:32.480676 21286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:32.480705 21286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:32.482512 21286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:38277,127.28.25.61:41591,127.28.25.60:46177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab
--principal=kudu/127.28.25.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.61:41591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.25.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:32.482868 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:32.483179 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:20:32.484390 21277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } attempt: 2
I20260811 11:20:32.484941 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.484761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41591 (local address 127.0.0.1:57320)
0811 11:20:32.484844 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.484899 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 11:20:32.485353 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.485241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41591 (local address 127.0.0.1:57336)
0811 11:20:32.485293 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.485328 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 11:20:32.485495 21277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.61:41591: Network error: Client connection negotiation failed: client connection to 127.28.25.61:41591: connect: Connection refused (error 111)
W20260811 11:20:32.489522 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:32.489507 21296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:32.489511 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:32.489619 21286 server_base.cc:1034] running on GCE node
I20260811 11:20:32.490083 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:32.490410 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:32.491595 21286 hybrid_clock.cc:648] HybridClock initialized: now 1786447232491567 us; error 60 us; skew 500 ppm
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:32.494463 21286 init.cc:377] Logged in from keytab as kudu/127.28.25.61@KRBTEST.COM (short username kudu)
I20260811 11:20:32.495081 21286 webserver.cc:460] Webserver started at http://127.28.25.61:42037/ using document root <none> and password file <none>
I20260811 11:20:32.495329 21286 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:32.495400 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:32.495566 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:32.496809 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/data/instance:
uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5"
format_stamp: "Formatted at 2026-08-11 11:20:32 on dist-test-slave-2w3w"
server_key: "db0017c4df118fce2be062b9dd3786c2"
server_key_iv: "7cd865d577829a6cc185a38e42527533"
server_key_version: "encryptionkey@0"
I20260811 11:20:32.497234 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/wal/instance:
uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5"
format_stamp: "Formatted at 2026-08-11 11:20:32 on dist-test-slave-2w3w"
server_key: "db0017c4df118fce2be062b9dd3786c2"
server_key_iv: "7cd865d577829a6cc185a38e42527533"
server_key_version: "encryptionkey@0"
I20260811 11:20:32.499094 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260811 11:20:32.500142 21303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:32.500392 21286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:32.500526 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/wal
uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5"
format_stamp: "Formatted at 2026-08-11 11:20:32 on dist-test-slave-2w3w"
server_key: "db0017c4df118fce2be062b9dd3786c2"
server_key_iv: "7cd865d577829a6cc185a38e42527533"
server_key_version: "encryptionkey@0"
I20260811 11:20:32.500622 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:32.521381 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:32.523896 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:32.524091 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:32.528659 21286 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.61:41591
I20260811 11:20:32.528700 21359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.61:41591 every 8 connection(s)
I20260811 11:20:32.529105 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
I20260811 11:20:32.530161 21360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:32.531857 21360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:32.538494 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 21286
I20260811 11:20:32.538641 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-1/wal/instance
I20260811 11:20:32.538877 28772 external_mini_cluster.cc:1428] Setting key f12a3deef53ba5e401ca4893f71dace8
I20260811 11:20:32.542308 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.532729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.28.25.62:38277)
0811 11:20:32.532842 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.532844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.533080 (+ 236us) server_negotiation.cc:407] Connection header received
0811 11:20:32.533568 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.533569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.533582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.533642 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.534463 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.535090 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.536108 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.536324 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.536389 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.539160 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.539162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.539163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.539172 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.540728 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.541217 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.541218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.541219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.541284 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.541581 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.541582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.541583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.541719 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.541797 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.542111 (+ 314us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.542208 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 11:20:32.542538 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.532691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:38277 (local address 127.0.0.1:40868)
0811 11:20:32.532935 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.532976 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:32.533475 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:32.533713 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:32.533716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:32.534134 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:32.534347 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.534352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.535174 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.535175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.536019 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.536022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.536445 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.536446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.536559 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.537197 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:32.537207 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:32.539024 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:32.540834 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.540854 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.540857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.541126 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.541374 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.541375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.541375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.541509 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.541801 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:32.541808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:32.541972 (+ 164us) client_negotiation.cc:769] Sending connection context
0811 11:20:32.542091 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 11:20:32.542272 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":131,"threads_started":1}
I20260811 11:20:32.543263 21360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.25.60@KRBTEST.COM; defaulting to no policy
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.60@KRBTEST.COM" created.
I20260811 11:20:32.551781 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.543595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41591 (local address 127.0.0.1:57338)
0811 11:20:32.543653 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.543664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:32.543708 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:32.543985 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:32.543986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:32.544144 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:32.544204 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.544205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.545117 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.545118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.545646 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.545647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.546167 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.546168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.546237 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.546806 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:32.546811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:32.548327 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:32.550330 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.550331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.550332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.550548 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.550796 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.550797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.550798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.550881 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.551421 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:32.551422 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:32.551485 (+ 63us) client_negotiation.cc:769] Sending connection context
0811 11:20:32.551581 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 11:20:32.551691 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 11:20:32.551975 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.543683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.28.25.61:41591)
0811 11:20:32.543847 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.543850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.543867 (+ 17us) server_negotiation.cc:407] Connection header received
0811 11:20:32.543882 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.543884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.543894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.543984 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.544290 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.545013 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.545747 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.546085 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.546171 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.548446 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.548458 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.548458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.548465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.550166 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.550629 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.550633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.550633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.550695 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.551034 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.551036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.551036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.551316 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.551416 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.551636 (+ 220us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.551774 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260811 11:20:32.552594 21360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:20:32.553068 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.552886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:41994)
0811 11:20:32.552990 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.553043 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260811 11:20:32.553272 21306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
I20260811 11:20:32.553862 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.553731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42000)
0811 11:20:32.553804 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.553838 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260811 11:20:32.554013 21360 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:46177: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab.
Entry for principal kudu/127.28.25.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab.
I20260811 11:20:32.571406 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.60:46177
--webserver_interface=127.28.25.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab
--principal=kudu/127.28.25.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:38277,127.28.25.61:41591,127.28.25.60:46177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:20:32.576684 21277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } attempt: 3
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
I20260811 11:20:32.586076 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.577185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41591 (local address 127.0.0.1:57342)
0811 11:20:32.577253 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.577277 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:32.577366 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:32.577556 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:32.577557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:32.577745 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:32.577891 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.577894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.578737 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.578738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.579411 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.579414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.579897 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.579898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.579996 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.580617 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:32.580622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:32.582424 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:32.584870 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.584872 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.584872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.585095 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.585331 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.585331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.585332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.585408 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.585719 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:32.585720 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:32.585774 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 11:20:32.585847 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 11:20:32.586002 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 11:20:32.586123 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.577263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.28.25.61:41591)
0811 11:20:32.577360 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.577363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.577377 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:20:32.577430 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.577431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.577453 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.577525 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.577998 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.578593 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.579513 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.579805 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.579899 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.582543 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.582546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.582546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.582556 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.584527 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.585184 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.585185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.585186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.585256 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.585471 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.585472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.585473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.585623 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.585694 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.585887 (+ 193us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.585987 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 11:20:32.586859 21277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:20:32.587333 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.587169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42002)
0811 11:20:32.587255 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.587303 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260811 11:20:32.587745 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.587634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42014)
0811 11:20:32.587686 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.587723 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:20:32.587909 21277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:46177: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
I20260811 11:20:32.603317 21360 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } attempt: 1
I20260811 11:20:32.603916 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.603712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42018)
0811 11:20:32.603786 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.603878 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260811 11:20:32.604290 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.604188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42020)
0811 11:20:32.604239 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.604272 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:20:32.604390 21360 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:46177: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
I20260811 11:20:32.619077 21277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } attempt: 1
I20260811 11:20:32.619688 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.619497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42032)
0811 11:20:32.619606 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.619654 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260811 11:20:32.620205 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.620108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42044)
0811 11:20:32.620157 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.620186 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:20:32.620326 21277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:46177: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
W20260811 11:20:32.662264 21368 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:20:32.662568 21368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:32.662601 21368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:32.662649 21368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:32.663938 21368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:32.664026 21368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:32.664050 21368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:32.664080 21368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:32.664126 21368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:32.665881 21368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:38277,127.28.25.61:41591,127.28.25.60:46177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab
--principal=kudu/127.28.25.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.60:46177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.25.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:32.666184 21368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:32.666440 21368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:20:32.672534 21360 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } attempt: 2
W20260811 11:20:32.672641 21375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:32.672648 21374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:32.672648 21377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:32.673144 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.673007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42052)
0811 11:20:32.673064 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.673098 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260811 11:20:32.673430 21368 server_base.cc:1034] running on GCE node
I20260811 11:20:32.673527 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.673437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42068)
0811 11:20:32.673478 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.673501 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260811 11:20:32.673673 21360 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:46177: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
I20260811 11:20:32.673751 21368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:32.674077 21368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:32.675259 21368 hybrid_clock.cc:648] HybridClock initialized: now 1786447232675242 us; error 26 us; skew 500 ppm
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:32.678128 21368 init.cc:377] Logged in from keytab as kudu/127.28.25.60@KRBTEST.COM (short username kudu)
I20260811 11:20:32.678632 21368 webserver.cc:460] Webserver started at http://127.28.25.60:38523/ using document root <none> and password file <none>
I20260811 11:20:32.678901 21368 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:32.679000 21368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:32.679159 21368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:32.680279 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/data/instance:
uuid: "c31e61b0d27040d3bfc8889eb31f606b"
format_stamp: "Formatted at 2026-08-11 11:20:32 on dist-test-slave-2w3w"
server_key: "5275ae7a6906697b35a16c144e6c775e"
server_key_iv: "b2b991fb9caf981e1952397c25e929f1"
server_key_version: "encryptionkey@0"
I20260811 11:20:32.680663 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/wal/instance:
uuid: "c31e61b0d27040d3bfc8889eb31f606b"
format_stamp: "Formatted at 2026-08-11 11:20:32 on dist-test-slave-2w3w"
server_key: "5275ae7a6906697b35a16c144e6c775e"
server_key_iv: "b2b991fb9caf981e1952397c25e929f1"
server_key_version: "encryptionkey@0"
I20260811 11:20:32.682122 21368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:20:32.683038 21385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:32.683220 21368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:32.683321 21368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/wal
uuid: "c31e61b0d27040d3bfc8889eb31f606b"
format_stamp: "Formatted at 2026-08-11 11:20:32 on dist-test-slave-2w3w"
server_key: "5275ae7a6906697b35a16c144e6c775e"
server_key_iv: "b2b991fb9caf981e1952397c25e929f1"
server_key_version: "encryptionkey@0"
I20260811 11:20:32.683413 21368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:32.695492 21277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } attempt: 2
I20260811 11:20:32.696205 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.696016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42076)
0811 11:20:32.696114 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.696162 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260811 11:20:32.696791 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.696679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42090)
0811 11:20:32.696732 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.696766 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260811 11:20:32.696941 21277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:46177: Network error: Client connection negotiation failed: client connection to 127.28.25.60:46177: connect: Connection refused (error 111)
I20260811 11:20:32.697657 21368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:32.700435 21368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:32.700628 21368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:32.705803 21368 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.60:46177
I20260811 11:20:32.705994 21445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.60:46177 every 8 connection(s)
I20260811 11:20:32.706306 21368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
I20260811 11:20:32.707387 21446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:32.708509 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 21368
I20260811 11:20:32.708612 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786446911996770-28772-0/minicluster-data/master-2/wal/instance
I20260811 11:20:32.708904 28772 external_mini_cluster.cc:1428] Setting key 785f8450432c43511f8b463e64465d74
I20260811 11:20:32.709527 21446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
I20260811 11:20:32.719946 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.710387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.28.25.62:38277)
0811 11:20:32.710446 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.710449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.710707 (+ 258us) server_negotiation.cc:407] Connection header received
0811 11:20:32.710807 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.710809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.710822 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.710881 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.711822 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.712318 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.713249 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.713468 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.713555 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.716362 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.716365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.716365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.716381 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.718253 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.718854 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.718855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.718856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.718924 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.719223 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.719224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.719225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.719371 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.719455 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.719748 (+ 293us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.719843 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 11:20:32.720173 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.710309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:38277 (local address 127.0.0.1:40878)
0811 11:20:32.710486 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.710599 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:32.710729 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:32.710917 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:32.710931 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:32.711333 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:32.711530 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.711535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.712430 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.712431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.713143 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.713146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.713594 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.713596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.713671 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.714364 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:32.714373 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:32.716231 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:32.718414 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.718422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.718424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.718730 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.719018 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.719019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.719019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.719146 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.719448 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:32.719455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:32.719610 (+ 155us) client_negotiation.cc:769] Sending connection context
0811 11:20:32.719727 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 11:20:32.719910 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260811 11:20:32.720695 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.709566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.28.25.60:46177)
0811 11:20:32.709801 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.709812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.709863 (+ 51us) server_negotiation.cc:407] Connection header received
0811 11:20:32.710430 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.710440 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.710515 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.710669 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.711445 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.712302 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.713109 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.713464 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.713595 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.716873 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.716884 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.716884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.716893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.718837 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.719356 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.719359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.719359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.719419 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.719656 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.719657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.719657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.719838 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.720408 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.720468 (+ 60us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.720573 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":111,"threads_started":1}
I20260811 11:20:32.720996 21446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:32.729214 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.721353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.28.25.61:41591)
0811 11:20:32.721403 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.721406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.721423 (+ 17us) server_negotiation.cc:407] Connection header received
0811 11:20:32.721468 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.721469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.721480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.721541 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.721937 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.722609 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.723653 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.724001 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.724098 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.726383 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.726384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.726385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.726392 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.727947 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.728364 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.728366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.728366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.728434 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.728631 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.728632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.728633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.728794 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.728873 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.729060 (+ 187us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.729140 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:20:32.729290 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.721269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:41591 (local address 127.0.0.1:57346)
0811 11:20:32.721327 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.721337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:32.721374 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:32.721560 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:32.721561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:32.721737 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:32.721809 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.721811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.722692 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.722693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.723552 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.723554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.724110 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.724111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.724222 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.724675 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:32.724680 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:32.726259 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:32.728050 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.728051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.728052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.728278 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.728511 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.728511 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.728512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.728566 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.728886 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:32.728887 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:32.728941 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 11:20:32.729007 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 11:20:32.729161 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260811 11:20:32.729806 21446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:20:32.730412 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.721525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.28.25.62:38277)
0811 11:20:32.721565 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.721567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.721574 (+ 7us) server_negotiation.cc:407] Connection header received
0811 11:20:32.721595 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.721596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.721605 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.721665 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.722233 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.722862 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.723820 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.724105 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.724176 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.727119 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.727121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.727121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.727129 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.728967 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.729417 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.729425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.729425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.729515 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.729798 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.729799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.729799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.729968 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.730165 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.730207 (+ 42us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.730299 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
I20260811 11:20:32.737877 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.730174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.28.25.60:46177)
0811 11:20:32.730216 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.730218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.730250 (+ 32us) server_negotiation.cc:407] Connection header received
0811 11:20:32.730313 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.730314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.730325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.730376 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.730689 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.731445 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.732249 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.732566 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.732634 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.734630 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.734632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.734632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.734639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.736451 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.736882 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.736884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.736884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.736955 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.737228 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.737229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.737230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.737370 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.737474 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.737643 (+ 169us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.737745 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 11:20:32.737982 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.730138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42108)
0811 11:20:32.730210 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.730221 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:32.730265 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:32.730395 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:32.730397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:32.730562 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:32.730610 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.730612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.731523 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.731524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.732162 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.732165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.732639 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.732640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.732692 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.733162 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:32.733168 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:32.734540 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:32.736596 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.736598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.736598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.736791 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.737035 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.737035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.737036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.737126 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.737465 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:32.737465 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:32.737512 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 11:20:32.737620 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 11:20:32.737858 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":135}
I20260811 11:20:32.740121 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.731175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.28.25.61:41591)
0811 11:20:32.731216 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.731218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.731226 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:20:32.731247 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.731247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.731256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.731318 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.731893 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.732518 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.733540 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.733752 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.733951 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.737110 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.737112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.737112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.737121 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.738825 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.739296 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.739298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.739298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.739364 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.739617 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.739618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.739618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.739747 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.739899 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.739988 (+ 89us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.740055 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:20:32.740108 21446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b: Bootstrap starting.
I20260811 11:20:32.741195 21446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:32.741564 21446 log.cc:824] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:32.741926 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:32.742365 21446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b: No bootstrap required, opened a new log
I20260811 11:20:32.744061 21446 raft_consensus.cc:348] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } }
I20260811 11:20:32.743471 21250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8906afc902b9413e9ea54fec80415b3e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40890
I20260811 11:20:32.744226 21446 raft_consensus.cc:729] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31e61b0d27040d3bfc8889eb31f606b, State: Initialized, Role: FOLLOWER
I20260811 11:20:32.744329 21446 consensus_queue.cc:260] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } }
I20260811 11:20:32.744678 21446 sys_catalog.cc:564] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:32.744665 21452 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } } }
I20260811 11:20:32.744793 21452 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [sys.catalog]: This master's current role is: FOLLOWER
W20260811 11:20:32.746665 21464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:32.746745 21464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:32.747195 21368 master_runner.cc:186] Error getting master registration for 127.28.25.62:38277: OK, instance_id { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" instance_seqno: 1786447232348014 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:20:32.750610 21368 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:20:32.764920 21360 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } attempt: 3
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
I20260811 11:20:32.774547 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.765468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.28.25.60:46177)
0811 11:20:32.765543 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.765545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.765556 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:20:32.765588 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.765589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.765600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.765683 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.766096 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.766927 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.768073 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.768404 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.768505 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.770877 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.770880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.770880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.770891 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.772849 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.773404 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.773406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.773407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.773487 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.773823 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.773824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.773824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.774064 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.774146 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.774338 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.774440 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:20:32.774561 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.765366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42118)
0811 11:20:32.765440 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.765454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:32.765507 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:32.765700 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:32.765701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:32.765919 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:32.765988 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.765991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.767084 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.767085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.767957 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.767961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.768511 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.768512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.768576 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.769146 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:32.769151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:32.770763 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:32.773008 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.773009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.773010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.773307 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.773637 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.773638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.773639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.773727 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.774150 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:32.774151 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:32.774212 (+ 61us) client_negotiation.cc:769] Sending connection context
0811 11:20:32.774289 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 11:20:32.774449 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 11:20:32.777130 21360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5: Bootstrap starting.
I20260811 11:20:32.778066 21360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:32.778457 21360 log.cc:824] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:32.779567 21360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5: No bootstrap required, opened a new log
I20260811 11:20:32.781826 21360 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } }
I20260811 11:20:32.782003 21360 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe84f9f1b90e483cb7a55dc99a38a2c5, State: Initialized, Role: FOLLOWER
I20260811 11:20:32.782085 21360 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } }
I20260811 11:20:32.782495 21360 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:32.782703 21467 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } } }
I20260811 11:20:32.782815 21467 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 11:20:32.784597 21480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:32.784740 21480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:32.784884 21286 master_runner.cc:186] Error getting master registration for 127.28.25.62:38277: OK, instance_id { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" instance_seqno: 1786447232348014 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:20:32.788416 21286 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:20:32.796129 21277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } attempt: 3
Aug 11 11:20:32 dist-test-slave-2w3w krb5kdc[21188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447232, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
I20260811 11:20:32.805816 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.796543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:46177 (local address 127.0.0.1:42128)
0811 11:20:32.796610 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:32.796621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:32.796666 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:32.796922 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:32.796923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:32.797144 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:32.797205 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.797207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.798210 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.798211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.799119 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:32.799124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.799932 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.799934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:32.800011 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.800636 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:32.800642 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:32.802387 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:32.804579 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.804580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.804581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.804874 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.805100 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:32.805100 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:32.805101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:32.805162 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:32.805468 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:32.805469 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:32.805520 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 11:20:32.805585 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:20:32.805704 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 11:20:32.806412 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.796681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.28.25.60:46177)
0811 11:20:32.796752 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.796754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.796764 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:20:32.796792 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.796794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.796806 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.796861 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.797317 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.798099 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.799390 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.799818 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.799914 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.802518 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.802521 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.802521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.802531 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.804465 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.804963 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.804965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.804966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.805029 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.805226 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.805227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.805228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.805371 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.805429 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.806191 (+ 762us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.806315 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:20:32.808619 21277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e: Bootstrap starting.
I20260811 11:20:32.809438 21277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:32.809782 21277 log.cc:824] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:32.810734 21277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e: No bootstrap required, opened a new log
I20260811 11:20:32.812363 21277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } }
I20260811 11:20:32.812553 21277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8906afc902b9413e9ea54fec80415b3e, State: Initialized, Role: FOLLOWER
I20260811 11:20:32.812645 21277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } }
I20260811 11:20:32.813009 21277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:32.813072 21250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 11:20:32.813145 21250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:32.814054 21250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } }
I20260811 11:20:32.814307 21250 leader_election.cc:288] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [CANDIDATE]: Term 1 election: Requested vote from peers fe84f9f1b90e483cb7a55dc99a38a2c5 (127.28.25.61:41591), c31e61b0d27040d3bfc8889eb31f606b (127.28.25.60:46177)
I20260811 11:20:32.814795 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8906afc902b9413e9ea54fec80415b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5"
I20260811 11:20:32.815675 21416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8906afc902b9413e9ea54fec80415b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c31e61b0d27040d3bfc8889eb31f606b"
I20260811 11:20:32.815807 21416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:32.813024 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } } }
I20260811 11:20:32.816045 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:32.816766 21416 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8906afc902b9413e9ea54fec80415b3e in term 1.
W20260811 11:20:32.817008 21495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260811 11:20:32.817099 21220 leader_election.cc:302] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8906afc902b9413e9ea54fec80415b3e, c31e61b0d27040d3bfc8889eb31f606b; no voters:
I20260811 11:20:32.817243 21483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:32.817387 21483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [term 1 LEADER]: Becoming Leader. State: Replica: 8906afc902b9413e9ea54fec80415b3e, State: Running, Role: LEADER
I20260811 11:20:32.817492 21483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } }
I20260811 11:20:32.817934 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8906afc902b9413e9ea54fec80415b3e. Latest consensus state: current_term: 1 leader_uuid: "8906afc902b9413e9ea54fec80415b3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } } }
I20260811 11:20:32.818019 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [sys.catalog]: This master's current role is: LEADER
W20260811 11:20:32.817117 21495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:32.818487 21333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:32.819713 21333 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8906afc902b9413e9ea54fec80415b3e in term 1.
I20260811 11:20:32.822116 21202 master_runner.cc:263] No leader master found from master 8906afc902b9413e9ea54fec80415b3e
I20260811 11:20:32.822194 21202 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:20:32.822928 21504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:32.823263 21504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:32.830425 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.819245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.28.25.62:38277)
0811 11:20:32.819291 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.819294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.822204 (+ 2910us) server_negotiation.cc:407] Connection header received
0811 11:20:32.822308 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.822309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.822320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.822350 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.823292 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.824062 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.825151 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.825467 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.825510 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.827289 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.827291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.827291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.827300 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.829053 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.829503 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.829504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.829505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.829564 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.829763 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.829763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.829764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.829902 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.829993 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.830226 (+ 233us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.830320 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 11:20:32.832093 21333 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [term 1 FOLLOWER]: Refusing update from remote peer 8906afc902b9413e9ea54fec80415b3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:32.832093 21416 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [term 1 FOLLOWER]: Refusing update from remote peer 8906afc902b9413e9ea54fec80415b3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:32.832487 21483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:32.832674 21483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:32.834383 21452 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8906afc902b9413e9ea54fec80415b3e. Latest consensus state: current_term: 1 leader_uuid: "8906afc902b9413e9ea54fec80415b3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } } }
I20260811 11:20:32.834504 21452 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:32.835644 21467 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8906afc902b9413e9ea54fec80415b3e. Latest consensus state: current_term: 1 leader_uuid: "8906afc902b9413e9ea54fec80415b3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } } }
I20260811 11:20:32.835757 21467 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:32.843232 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.821934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42136 (local address 127.28.25.60:46177)
0811 11:20:32.821974 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.821976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.831376 (+ 9400us) server_negotiation.cc:407] Connection header received
0811 11:20:32.831497 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.831498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.831513 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.831544 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.835952 (+ 4408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.836689 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.837717 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.838033 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.838078 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.839870 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.839871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.839872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.839881 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.841744 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.842175 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.842176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.842177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.842240 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.842490 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.842491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.842491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.842651 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.842743 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.843009 (+ 266us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.843099 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 11:20:32.844569 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:32.821551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.28.25.61:41591)
0811 11:20:32.821596 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.821598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.831121 (+ 9523us) server_negotiation.cc:407] Connection header received
0811 11:20:32.831237 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.831239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.831253 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.831286 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.833295 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.834008 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.839056 (+ 5048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:32.839411 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:32.839462 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:32.841341 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:32.841343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:32.841343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:32.841352 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:32.843144 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.843571 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.843572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.843573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.843641 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:32.843883 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:32.843884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:32.843884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:32.844043 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:32.844117 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:32.844327 (+ 210us) server_negotiation.cc:299] Negotiation successful
0811 11:20:32.844434 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260811 11:20:32.844592 21225 tablet.cc:2369] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:20:32.846204 21502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8906afc902b9413e9ea54fec80415b3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } } }
I20260811 11:20:32.846352 21502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:32.847227 21452 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8906afc902b9413e9ea54fec80415b3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } } }
I20260811 11:20:32.847347 21452 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:32.847702 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8906afc902b9413e9ea54fec80415b3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } } }
I20260811 11:20:32.847815 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:32.848356 21467 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8906afc902b9413e9ea54fec80415b3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8906afc902b9413e9ea54fec80415b3e" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 38277 } } peers { permanent_uuid: "fe84f9f1b90e483cb7a55dc99a38a2c5" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 41591 } } peers { permanent_uuid: "c31e61b0d27040d3bfc8889eb31f606b" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 46177 } } }
I20260811 11:20:32.848456 21467 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:32.849105 21509 mvcc.cc:205] Tried to move back new op lower bound from 7317287865676681216 to 7317287865621016576. Current Snapshot: MvccSnapshot[applied={T|T < 7317287865676681216}]
I20260811 11:20:32.851660 21504 catalog_manager.cc:1347] Generated new cluster ID: 717faf81086f4012ade02ae860aac74a
I20260811 11:20:32.851744 21504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:32.870401 21504 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:32.871250 21504 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:32.876919 21504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8906afc902b9413e9ea54fec80415b3e: Generated new TSK 0
I20260811 11:20:32.877172 21504 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 11:20:32.909080 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:32.907235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.28.25.61:41591)
0811 11:20:32.907283 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.907285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.908314 (+ 1029us) server_negotiation.cc:407] Connection header received
0811 11:20:32.908442 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.908443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.908456 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.908489 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.909004 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57374: BlockingRecv error: recv got EOF from 127.0.0.1:57374 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260811 11:20:32.909965 21449 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:32.907612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.28.25.60:46177)
0811 11:20:32.907658 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.907660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.909404 (+ 1744us) server_negotiation.cc:407] Connection header received
0811 11:20:32.909512 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.909513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.909525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.909555 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.909902 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42140: BlockingRecv error: recv got EOF from 127.0.0.1:42140 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260811 11:20:32.910813 21280 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:32.908003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.28.25.62:38277)
0811 11:20:32.908044 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.908047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.910194 (+ 2147us) server_negotiation.cc:407] Connection header received
0811 11:20:32.910298 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.910299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.910312 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.910341 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.910750 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40912: BlockingRecv error: recv got EOF from 127.0.0.1:40912 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260811 11:20:32.913836 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:32.912776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57376 (local address 127.28.25.61:41591)
0811 11:20:32.912821 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.912823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.913084 (+ 261us) server_negotiation.cc:407] Connection header received
0811 11:20:32.913201 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.913202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.913224 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.913255 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.913781 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57376: BlockingRecv error: recv got EOF from 127.0.0.1:57376 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260811 11:20:32.917093 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:32.916200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.28.25.61:41591)
0811 11:20:32.916240 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.916242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.916529 (+ 287us) server_negotiation.cc:407] Connection header received
0811 11:20:32.916632 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.916633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.916644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.916675 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.917036 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57388: BlockingRecv error: recv got EOF from 127.0.0.1:57388 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260811 11:20:32.921200 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:32.920329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.28.25.61:41591)
0811 11:20:32.920384 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.920386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.920653 (+ 267us) server_negotiation.cc:407] Connection header received
0811 11:20:32.920752 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.920753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.920764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.920794 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.921140 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57400: BlockingRecv error: recv got EOF from 127.0.0.1:57400 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260811 11:20:32.927582 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:32.926520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.28.25.61:41591)
0811 11:20:32.926573 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.926575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.926905 (+ 330us) server_negotiation.cc:407] Connection header received
0811 11:20:32.927062 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.927063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.927077 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.927108 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.927521 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57408: BlockingRecv error: recv got EOF from 127.0.0.1:57408 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260811 11:20:32.937920 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:32.936918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57414 (local address 127.28.25.61:41591)
0811 11:20:32.936971 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.936973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.937275 (+ 302us) server_negotiation.cc:407] Connection header received
0811 11:20:32.937390 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.937391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.937404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.937435 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.937847 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57414: BlockingRecv error: recv got EOF from 127.0.0.1:57414 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260811 11:20:32.956333 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:32.955482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.28.25.61:41591)
0811 11:20:32.955535 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.955538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.955711 (+ 173us) server_negotiation.cc:407] Connection header received
0811 11:20:32.955787 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.955788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.955801 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.955881 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.956248 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57426: BlockingRecv error: recv got EOF from 127.0.0.1:57426 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260811 11:20:32.990401 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:32.989517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.28.25.61:41591)
0811 11:20:32.989573 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:32.989575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:32.989782 (+ 207us) server_negotiation.cc:407] Connection header received
0811 11:20:32.989867 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:32.989868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:32.989882 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:32.989958 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:32.990331 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57428: BlockingRecv error: recv got EOF from 127.0.0.1:57428 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260811 11:20:33.056480 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:33.055770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.28.25.61:41591)
0811 11:20:33.055853 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:33.055855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:33.055940 (+ 85us) server_negotiation.cc:407] Connection header received
0811 11:20:33.056002 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:33.056003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:33.056015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:33.056077 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:33.056405 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57434: BlockingRecv error: recv got EOF from 127.0.0.1:57434 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260811 11:20:33.186748 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:33.185926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57438 (local address 127.28.25.61:41591)
0811 11:20:33.186019 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:33.186022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:33.186034 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:20:33.186074 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:33.186075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:33.186085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:33.186172 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:33.186669 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57438: BlockingRecv error: recv got EOF from 127.0.0.1:57438 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260811 11:20:33.445271 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:33.444299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.28.25.61:41591)
0811 11:20:33.444421 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:33.444424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:33.444503 (+ 79us) server_negotiation.cc:407] Connection header received
0811 11:20:33.444600 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:33.444601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:33.444621 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:33.444685 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:33.445189 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57440: BlockingRecv error: recv got EOF from 127.0.0.1:57440 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260811 11:20:33.747306 21464 catalog_manager.cc:1259] Loaded cluster ID: 717faf81086f4012ade02ae860aac74a
I20260811 11:20:33.747383 21464 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b: loading cluster ID for follower catalog manager: success
I20260811 11:20:33.748239 21464 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b: acquiring CA information for follower catalog manager: success
I20260811 11:20:33.748580 21464 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c31e61b0d27040d3bfc8889eb31f606b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:20:33.785272 21480 catalog_manager.cc:1259] Loaded cluster ID: 717faf81086f4012ade02ae860aac74a
I20260811 11:20:33.785356 21480 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5: loading cluster ID for follower catalog manager: success
I20260811 11:20:33.786254 21480 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5: acquiring CA information for follower catalog manager: success
I20260811 11:20:33.786589 21480 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe84f9f1b90e483cb7a55dc99a38a2c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:20:33.961866 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:33.958884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.28.25.61:41591)
0811 11:20:33.959050 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:33.959051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:33.959066 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:20:33.959153 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:33.959155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:33.959207 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:33.959326 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:33.959586 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:33.960159 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:33.960974 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:33.961184 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:33.961250 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:33.961460 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:33.961584 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:33.961646 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:33.961694 (+ 48us) server_negotiation.cc:299] Negotiation successful
0811 11:20:33.961734 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":59,"threads_started":1}
I20260811 11:20:33.967592 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:33.964384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.28.25.60:46177)
0811 11:20:33.964539 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:33.964541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:33.964550 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:20:33.964591 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:33.964592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:33.964639 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:33.964952 (+ 313us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:33.965102 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:33.965665 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:33.966537 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:33.966854 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:33.966927 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:33.967211 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:33.967355 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:33.967463 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:33.967479 (+ 16us) server_negotiation.cc:299] Negotiation successful
0811 11:20:33.967504 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260811 11:20:33.973035 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:33.970013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.28.25.60:46177)
0811 11:20:33.970068 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:33.970070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:33.970251 (+ 181us) server_negotiation.cc:407] Connection header received
0811 11:20:33.970378 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:33.970379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:33.970390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:33.970438 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:33.970584 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:33.971282 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:33.972178 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:33.972486 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:33.972546 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:33.972732 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:33.972774 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:33.972833 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:33.972906 (+ 73us) server_negotiation.cc:299] Negotiation successful
0811 11:20:33.972938 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 11:20:33.973241 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:33.969639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.28.25.61:41591)
0811 11:20:33.969713 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:33.969716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:33.969837 (+ 121us) server_negotiation.cc:407] Connection header received
0811 11:20:33.969951 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:33.969952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:33.969967 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:33.969998 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:33.970234 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:33.971084 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:33.972322 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:33.972636 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:33.972700 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:33.972918 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:33.972954 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:33.972984 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:33.973113 (+ 129us) server_negotiation.cc:299] Negotiation successful
0811 11:20:33.973143 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:20:33.973824 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:33.969646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.28.25.62:38277)
0811 11:20:33.969816 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:33.969819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:33.969913 (+ 94us) server_negotiation.cc:407] Connection header received
0811 11:20:33.970027 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:33.970028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:33.970095 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:33.970180 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:33.970414 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:33.971331 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:33.972544 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:33.972904 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:33.972970 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:33.973262 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:33.973401 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:33.973475 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:33.973701 (+ 226us) server_negotiation.cc:299] Negotiation successful
0811 11:20:33.973734 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260811 11:20:33.975886 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 21202
I20260811 11:20:33.983757 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 21286
I20260811 11:20:33.992460 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 21368
2026-08-11T11:20:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3039 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:20:34 dist-test-slave-2w3w krb5kdc[21609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:20:34 dist-test-slave-2w3w krb5kdc[21609](info): set up 2 sockets
Aug 11 11:20:34 dist-test-slave-2w3w krb5kdc[21609](info): commencing operation
krb5kdc: starting...
W20260811 11:20:35.105990 28772 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/test-user.keytab.
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:20:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:20:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.25.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
Entry for principal kudu/127.28.25.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab.
I20260811 11:20:35.246428 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:45251
--webserver_interface=127.28.25.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:45251,127.28.25.61:35735,127.28.25.60:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:20:35.331561 21624 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:20:35.331791 21624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:35.331809 21624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:35.331831 21624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:35.333038 21624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:35.333089 21624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:35.333104 21624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:35.333117 21624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:35.333129 21624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:35.334765 21624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:45251,127.28.25.61:35735,127.28.25.60:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.62.keytab
--principal=kudu/127.28.25.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.62:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.25.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:35.335083 21624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:35.335350 21624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:20:35.340489 21631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:35.340608 21624 server_base.cc:1034] running on GCE node
W20260811 11:20:35.340560 21632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:35.340493 21634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:35.341064 21624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:35.341390 21624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:35.342579 21624 hybrid_clock.cc:648] HybridClock initialized: now 1786447235342573 us; error 39 us; skew 500 ppm
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:35.350778 21624 init.cc:377] Logged in from keytab as kudu/127.28.25.62@KRBTEST.COM (short username kudu)
I20260811 11:20:35.351303 21624 webserver.cc:460] Webserver started at http://127.28.25.62:38979/ using document root <none> and password file <none>
I20260811 11:20:35.351568 21624 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:35.351619 21624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:35.351753 21624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:35.352893 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/data/instance:
uuid: "e840edae744743868867b853c93e7ae3"
format_stamp: "Formatted at 2026-08-11 11:20:35 on dist-test-slave-2w3w"
server_key: "4994f8d2612d0c3dc4ee12fc4a64682c"
server_key_iv: "d73ec413ebc953d5badec6b648bf9d5d"
server_key_version: "encryptionkey@0"
I20260811 11:20:35.353276 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/wal/instance:
uuid: "e840edae744743868867b853c93e7ae3"
format_stamp: "Formatted at 2026-08-11 11:20:35 on dist-test-slave-2w3w"
server_key: "4994f8d2612d0c3dc4ee12fc4a64682c"
server_key_iv: "d73ec413ebc953d5badec6b648bf9d5d"
server_key_version: "encryptionkey@0"
I20260811 11:20:35.354724 21624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 11:20:35.355600 21641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:35.355875 21624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 11:20:35.355994 21624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/wal
uuid: "e840edae744743868867b853c93e7ae3"
format_stamp: "Formatted at 2026-08-11 11:20:35 on dist-test-slave-2w3w"
server_key: "4994f8d2612d0c3dc4ee12fc4a64682c"
server_key_iv: "d73ec413ebc953d5badec6b648bf9d5d"
server_key_version: "encryptionkey@0"
I20260811 11:20:35.356081 21624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:35.382192 21624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:35.385000 21624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:35.385138 21624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:35.390311 21624 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.62:45251
I20260811 11:20:35.390355 21699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.62:45251 every 8 connection(s)
I20260811 11:20:35.390798 21624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/data/info.pb
I20260811 11:20:35.392019 21701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:35.392961 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 21624
I20260811 11:20:35.393081 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-0/wal/instance
I20260811 11:20:35.393281 28772 external_mini_cluster.cc:1428] Setting key 63bed2f84b072617eec438d6604e4206
I20260811 11:20:35.394726 21701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.25.61@KRBTEST.COM; defaulting to no policy
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.61@KRBTEST.COM" created.
I20260811 11:20:35.406646 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.395769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:45251 (local address 127.0.0.1:60700)
0811 11:20:35.396005 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.396062 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:35.396654 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:35.396987 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:35.396990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:35.397308 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:35.397497 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.397502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.398683 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.398685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.399547 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.399551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.400178 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.400179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.400253 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.400962 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:35.400972 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:35.402657 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:35.404742 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.404751 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.404753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.404980 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.405273 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.405274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.405275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.405391 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.405925 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:35.405933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:35.406025 (+ 92us) client_negotiation.cc:769] Sending connection context
0811 11:20:35.406136 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 11:20:35.406341 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":120,"threads_started":1}
I20260811 11:20:35.406654 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.395803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.28.25.62:45251)
0811 11:20:35.396062 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.396073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.396197 (+ 124us) server_negotiation.cc:407] Connection header received
0811 11:20:35.396835 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.396837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.396860 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.396982 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.397620 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.398570 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.399666 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.400035 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.400184 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.402752 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.402761 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.402762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.402770 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.404635 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.405085 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.405088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.405088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.405182 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.405470 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.405471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.405471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.405831 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.405921 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.406211 (+ 290us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.406430 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":230,"spinlock_wait_cycles":11136,"thread_start_us":132,"threads_started":1}
I20260811 11:20:35.407500 21701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:20:35.407934 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.407793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:35735 (local address 127.0.0.1:40500)
0811 11:20:35.407857 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.407908 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:20:35.408090 21642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.25.61:35735: connect: Connection refused (error 111)
I20260811 11:20:35.408599 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.408498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:35735 (local address 127.0.0.1:40506)
0811 11:20:35.408545 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.408579 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260811 11:20:35.408705 21701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.61:35735: Network error: Client connection negotiation failed: client connection to 127.28.25.61:35735: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab.
Entry for principal kudu/127.28.25.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab.
I20260811 11:20:35.425983 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.61:35735
--webserver_interface=127.28.25.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab
--principal=kudu/127.28.25.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:45251,127.28.25.61:35735,127.28.25.60:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:20:35.457969 21701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } attempt: 1
I20260811 11:20:35.458536 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.458368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:35735 (local address 127.0.0.1:40510)
0811 11:20:35.458468 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.458504 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 11:20:35.458906 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.458805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:35735 (local address 127.0.0.1:40526)
0811 11:20:35.458851 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.458879 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260811 11:20:35.459013 21701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.61:35735: Network error: Client connection negotiation failed: client connection to 127.28.25.61:35735: connect: Connection refused (error 111)
W20260811 11:20:35.513653 21710 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:20:35.513940 21710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:35.513973 21710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:35.514019 21710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:35.515275 21710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:35.515357 21710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:35.515380 21710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:35.515401 21710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:35.515432 21710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:35.517056 21710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:45251,127.28.25.61:35735,127.28.25.60:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.61.keytab
--principal=kudu/127.28.25.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.61:35735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.25.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:35.517367 21710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:35.517627 21710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:20:35.522979 21710 server_base.cc:1034] running on GCE node
W20260811 11:20:35.522933 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:35.522933 21719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:35.522921 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:35.523481 21710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:35.523927 21710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:35.525128 21710 hybrid_clock.cc:648] HybridClock initialized: now 1786447235525116 us; error 54 us; skew 500 ppm
I20260811 11:20:35.527151 21701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } attempt: 2
I20260811 11:20:35.527577 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.527439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:35735 (local address 127.0.0.1:40528)
0811 11:20:35.527510 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.527545 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:35.527992 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.527897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:35735 (local address 127.0.0.1:40536)
0811 11:20:35.527934 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.527966 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260811 11:20:35.528092 21701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.61:35735: Network error: Client connection negotiation failed: client connection to 127.28.25.61:35735: connect: Connection refused (error 111)
I20260811 11:20:35.528301 21710 init.cc:377] Logged in from keytab as kudu/127.28.25.61@KRBTEST.COM (short username kudu)
I20260811 11:20:35.528821 21710 webserver.cc:460] Webserver started at http://127.28.25.61:34395/ using document root <none> and password file <none>
I20260811 11:20:35.529099 21710 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:35.529145 21710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:35.529325 21710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:35.530454 21710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/data/instance:
uuid: "ff39084911744df68596a14c4682198b"
format_stamp: "Formatted at 2026-08-11 11:20:35 on dist-test-slave-2w3w"
server_key: "ec20d98003e4fa0a07df3bf6020de0e9"
server_key_iv: "c9b39cbeef964eac7750dfa331f0d2bf"
server_key_version: "encryptionkey@0"
I20260811 11:20:35.530833 21710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/wal/instance:
uuid: "ff39084911744df68596a14c4682198b"
format_stamp: "Formatted at 2026-08-11 11:20:35 on dist-test-slave-2w3w"
server_key: "ec20d98003e4fa0a07df3bf6020de0e9"
server_key_iv: "c9b39cbeef964eac7750dfa331f0d2bf"
server_key_version: "encryptionkey@0"
I20260811 11:20:35.532300 21710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:35.533191 21727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:35.533454 21710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:35.533537 21710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/wal
uuid: "ff39084911744df68596a14c4682198b"
format_stamp: "Formatted at 2026-08-11 11:20:35 on dist-test-slave-2w3w"
server_key: "ec20d98003e4fa0a07df3bf6020de0e9"
server_key_iv: "c9b39cbeef964eac7750dfa331f0d2bf"
server_key_version: "encryptionkey@0"
I20260811 11:20:35.533643 21710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:35.541744 21710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:35.544328 21710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:35.544509 21710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:35.549209 21710 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.61:35735
I20260811 11:20:35.549247 21783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.61:35735 every 8 connection(s)
I20260811 11:20:35.549659 21710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/data/info.pb
I20260811 11:20:35.550757 21784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:35.552536 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 21710
I20260811 11:20:35.552639 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-1/wal/instance
I20260811 11:20:35.552868 28772 external_mini_cluster.cc:1428] Setting key c60af3aa29ced0202df511dc2827cac3
I20260811 11:20:35.552858 21784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.25.60@KRBTEST.COM; defaulting to no policy
I20260811 11:20:35.563594 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.553838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60704 (local address 127.28.25.62:45251)
0811 11:20:35.553907 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.553909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.554155 (+ 246us) server_negotiation.cc:407] Connection header received
0811 11:20:35.554721 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.554723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.554737 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.554799 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.555581 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.556101 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.557139 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.557361 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.557442 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.559954 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.559956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.559957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.559967 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.561919 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.562398 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.562400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.562400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.562484 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.562818 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.562819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.562820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.563044 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.563112 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.563398 (+ 286us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.563515 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 11:20:35.563741 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.553792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:45251 (local address 127.0.0.1:60704)
0811 11:20:35.554017 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.554057 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:35.554597 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:35.554872 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:35.554875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:35.555229 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:35.555465 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.555470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.556202 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.556203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.557037 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.557040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.557470 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.557472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.557620 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.558189 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:35.558198 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:35.559796 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:35.562041 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.562050 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.562052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.562314 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.562567 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.562567 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.562568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.562741 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.563119 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:35.563126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:35.563262 (+ 136us) client_negotiation.cc:769] Sending connection context
0811 11:20:35.563361 (+ 99us) client_negotiation.cc:240] Negotiation successful
0811 11:20:35.563511 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":132,"threads_started":1}
I20260811 11:20:35.564384 21784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.25.60@KRBTEST.COM" created.
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
I20260811 11:20:35.573025 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.564734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:35735 (local address 127.0.0.1:40548)
0811 11:20:35.564788 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.564801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:35.564848 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:35.565226 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:35.565227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:35.565375 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:35.565428 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.565454 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.566637 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.566638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.567203 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.567205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.567749 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.567750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.567820 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.568292 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:35.568302 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:35.569790 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:35.571860 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.571861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.571861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.572044 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.572270 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.572271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.572271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.572340 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.572639 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:35.572640 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:35.572681 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 11:20:35.572747 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 11:20:35.572884 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":17}
I20260811 11:20:35.573037 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.564805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.28.25.61:35735)
0811 11:20:35.565054 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.565058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.565076 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:20:35.565093 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.565094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.565133 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.565213 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.565561 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.566538 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.567305 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.567668 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.567755 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.569933 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.569944 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.569945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.569956 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.571737 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.572143 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.572146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.572146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.572204 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.572404 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.572405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.572406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.572569 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.572665 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.572775 (+ 110us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.572936 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":222,"thread_start_us":104,"threads_started":1}
I20260811 11:20:35.573726 21784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:20:35.574185 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.574051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52350)
0811 11:20:35.574110 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.574161 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 11:20:35.574347 21729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
I20260811 11:20:35.575104 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.574912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52366)
0811 11:20:35.575037 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.575071 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260811 11:20:35.575289 21784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:45981: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.25.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab.
Entry for principal kudu/127.28.25.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab.
I20260811 11:20:35.587404 28772 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR10UZg/build/release/bin/kudu
/tmp/dist-test-taskR10UZg/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.60:45981
--webserver_interface=127.28.25.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab
--principal=kudu/127.28.25.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.25.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.25.62:45251,127.28.25.61:35735,127.28.25.60:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:20:35.619274 21701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } attempt: 3
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
I20260811 11:20:35.624533 21784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } attempt: 1
I20260811 11:20:35.625044 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.624833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52380)
0811 11:20:35.624911 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.625004 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260811 11:20:35.625530 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.625424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52388)
0811 11:20:35.625471 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.625497 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260811 11:20:35.625712 21784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:45981: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
I20260811 11:20:35.627127 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.619688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:35735 (local address 127.0.0.1:40556)
0811 11:20:35.619766 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.619782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:35.619866 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:35.620030 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:35.620031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:35.620213 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:35.620283 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.620285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.621005 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.621006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.621564 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.621566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.622023 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.622025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.622104 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.622621 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:35.622626 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:35.624017 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:35.625991 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.625992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.625993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.626165 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.626410 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.626411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.626411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.626455 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.626781 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:35.626781 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:35.626826 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 11:20:35.626881 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 11:20:35.627055 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 11:20:35.627187 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.619776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.28.25.61:35735)
0811 11:20:35.619857 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.619859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.619873 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:20:35.619935 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.619937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.619956 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.620021 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.620379 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.620909 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.621668 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.621906 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.621991 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.624126 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.624127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.624128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.624135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.625874 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.626249 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.626250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.626251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.626328 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.626519 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.626520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.626520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.626699 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.626792 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.626938 (+ 146us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.627058 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:20:35.628798 21701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:20:35.629165 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.629060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52396)
0811 11:20:35.629105 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.629137 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260811 11:20:35.629576 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.629480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52406)
0811 11:20:35.629528 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.629553 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260811 11:20:35.629698 21701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:45981: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
I20260811 11:20:35.660845 21701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } attempt: 1
I20260811 11:20:35.661369 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.661228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52420)
0811 11:20:35.661305 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.661338 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260811 11:20:35.661757 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.661656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52436)
0811 11:20:35.661704 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.661738 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260811 11:20:35.661856 21701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:45981: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
W20260811 11:20:35.677162 21791 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:20:35.677456 21791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:20:35.677500 21791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:20:35.677544 21791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:20:35.679124 21791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:20:35.679199 21791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:20:35.679231 21791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:20:35.679262 21791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:20:35.679289 21791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:20:35.681607 21791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.25.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.25.62:45251,127.28.25.61:35735,127.28.25.60:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/krb5kdc/kudu_127.28.25.60.keytab
--principal=kudu/127.28.25.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.25.60:45981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.25.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 10:16:46 UTC on ccycloud.quasar-tsnumj.root.comops.site
build id 4580137
I20260811 11:20:35.682039 21791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:20:35.682313 21791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:20:35.688321 21791 server_base.cc:1034] running on GCE node
W20260811 11:20:35.688223 21800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:35.688223 21797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:20:35.688241 21796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:20:35.688814 21791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:20:35.689142 21791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:20:35.690352 21791 hybrid_clock.cc:648] HybridClock initialized: now 1786447235690347 us; error 44 us; skew 500 ppm
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:20:35.693543 21791 init.cc:377] Logged in from keytab as kudu/127.28.25.60@KRBTEST.COM (short username kudu)
I20260811 11:20:35.693899 21784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } attempt: 2
I20260811 11:20:35.694190 21791 webserver.cc:460] Webserver started at http://127.28.25.60:38133/ using document root <none> and password file <none>
I20260811 11:20:35.694449 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.694289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52438)
0811 11:20:35.694363 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.694406 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260811 11:20:35.694473 21791 fs_manager.cc:359] Metadata directory not provided
I20260811 11:20:35.694542 21791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:20:35.694734 21791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:20:35.694857 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.694743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52440)
0811 11:20:35.694803 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.694832 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260811 11:20:35.695007 21784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.25.60:45981: Network error: Client connection negotiation failed: client connection to 127.28.25.60:45981: connect: Connection refused (error 111)
I20260811 11:20:35.695993 21791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/data/instance:
uuid: "cbeb147a460445eb8f54de49a9bfcecf"
format_stamp: "Formatted at 2026-08-11 11:20:35 on dist-test-slave-2w3w"
server_key: "83c721c3d3d6ba8f95532fcfa590e658"
server_key_iv: "545198f1b3ff3502faf7b43512ebf608"
server_key_version: "encryptionkey@0"
I20260811 11:20:35.696339 21791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/wal/instance:
uuid: "cbeb147a460445eb8f54de49a9bfcecf"
format_stamp: "Formatted at 2026-08-11 11:20:35 on dist-test-slave-2w3w"
server_key: "83c721c3d3d6ba8f95532fcfa590e658"
server_key_iv: "545198f1b3ff3502faf7b43512ebf608"
server_key_version: "encryptionkey@0"
I20260811 11:20:35.697721 21791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 11:20:35.698580 21807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:20:35.698825 21791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:20:35.698928 21791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/data,/tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/wal
uuid: "cbeb147a460445eb8f54de49a9bfcecf"
format_stamp: "Formatted at 2026-08-11 11:20:35 on dist-test-slave-2w3w"
server_key: "83c721c3d3d6ba8f95532fcfa590e658"
server_key_iv: "545198f1b3ff3502faf7b43512ebf608"
server_key_version: "encryptionkey@0"
I20260811 11:20:35.699043 21791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:20:35.712561 21791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:20:35.715121 21791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:20:35.715255 21791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:20:35.719825 21791 rpc_server.cc:307] RPC server started. Bound to: 127.28.25.60:45981
I20260811 11:20:35.719867 21866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.25.60:45981 every 8 connection(s)
I20260811 11:20:35.720268 21791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/data/info.pb
I20260811 11:20:35.721400 21867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:20:35.723137 21867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:20:35.723973 28772 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR10UZg/build/release/bin/kudu as pid 21791
I20260811 11:20:35.724100 28772 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR10UZg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786446911996770-28772-0/minicluster-data/master-2/wal/instance
I20260811 11:20:35.724327 28772 external_mini_cluster.cc:1428] Setting key a9ed0be9f9fc90a5bf7905e58fbacc72
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
I20260811 11:20:35.734754 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.723992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.28.25.62:45251)
0811 11:20:35.724087 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.724090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.724250 (+ 160us) server_negotiation.cc:407] Connection header received
0811 11:20:35.724848 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.724858 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.724871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.724941 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.725850 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.726653 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.727998 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.728245 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.728320 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.731113 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.731114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.731115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.731126 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.732795 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.733511 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.733513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.733513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.733588 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.733908 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.733909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.733910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.734137 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.734200 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.734536 (+ 336us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.734637 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 11:20:35.734982 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.723888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.62:45251 (local address 127.0.0.1:60706)
0811 11:20:35.724115 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.724160 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:35.724707 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:35.725035 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:35.725040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:35.725476 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:35.725738 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.725745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.726747 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.726748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.727903 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.727906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.728351 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.728353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.728492 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.729085 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:35.729096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:35.730872 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:35.732880 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.732887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.732889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.733395 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.733657 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.733658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.733658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.733769 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.734203 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:35.734211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:35.734391 (+ 180us) client_negotiation.cc:769] Sending connection context
0811 11:20:35.734497 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 11:20:35.734673 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":9,"thread_start_us":101,"threads_started":1}
I20260811 11:20:35.734982 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.724774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.28.25.60:45981)
0811 11:20:35.724955 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.724961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.725069 (+ 108us) server_negotiation.cc:407] Connection header received
0811 11:20:35.725113 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.725115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.725135 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.725203 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.726006 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.727133 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.728156 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.728577 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.728630 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.731466 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.731473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.731474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.731480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.733432 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.733835 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.733838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.733839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.733904 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.734140 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.734140 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.734141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.734404 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.734591 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.734675 (+ 84us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.734758 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
I20260811 11:20:35.735634 21867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:20:35.736974 21701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } attempt: 2
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.62@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.62@KRBTEST.COM
I20260811 11:20:35.744441 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.735916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.61:35735 (local address 127.0.0.1:40570)
0811 11:20:35.736074 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.736084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:35.736134 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:35.736270 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:35.736271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:35.736444 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:35.736488 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.736490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.737461 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.737462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.738061 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.738063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.738561 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.738562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.738633 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.739380 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:35.739385 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:35.740876 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:35.743004 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.743006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.743006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.743333 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.743618 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.743619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.743620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.743682 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.744050 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:35.744051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:35.744114 (+ 63us) client_negotiation.cc:769] Sending connection context
0811 11:20:35.744201 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 11:20:35.744310 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260811 11:20:35.744455 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.735993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.28.25.61:35735)
0811 11:20:35.736047 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.736049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.736126 (+ 77us) server_negotiation.cc:407] Connection header received
0811 11:20:35.736179 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.736180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.736193 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.736253 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.736547 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.737365 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.738148 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.738469 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.738538 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.740971 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.740972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.740973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.740981 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.742850 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.743452 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.743454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.743454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.743519 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.743766 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.743768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.743769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.743950 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.744034 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.744258 (+ 224us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.744351 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 11:20:35.745195 21867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:20:35.745524 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.737206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52460)
0811 11:20:35.737243 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.737255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:35.737293 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:35.737507 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:35.737508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:35.737660 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:35.737696 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.737698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.738614 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.738615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.739356 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.739358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.739733 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.739734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.739792 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.740208 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:35.740214 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:35.742062 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:35.744301 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.744302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.744302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.744519 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.744743 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.744744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.744744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.744800 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.745185 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:35.745186 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:35.745227 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 11:20:35.745300 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 11:20:35.745427 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260811 11:20:35.745937 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.736326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.28.25.62:45251)
0811 11:20:35.736372 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.736374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.736382 (+ 8us) server_negotiation.cc:407] Connection header received
0811 11:20:35.736402 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.736403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.736418 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.736477 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.737104 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.737957 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.738739 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.739039 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.739108 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.742418 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.742420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.742420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.742428 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.744302 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.744757 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.744759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.744759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.744823 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.745031 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.745032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.745032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.745168 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.745396 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.745736 (+ 340us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.745841 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 11:20:35.745967 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.737318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.28.25.60:45981)
0811 11:20:35.737356 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.737357 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.737364 (+ 7us) server_negotiation.cc:407] Connection header received
0811 11:20:35.737383 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.737383 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.737392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.737785 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.737815 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.738552 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.739450 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.739663 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.739734 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.742176 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.742178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.742179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.742188 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.744173 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.744612 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.744613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.744614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.744674 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.744864 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.744865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.744865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.744998 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.745090 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.745354 (+ 264us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.745830 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 11:20:35.748080 21701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3: Bootstrap starting.
I20260811 11:20:35.748886 21701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:35.749198 21701 log.cc:824] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3: Log is configured to *not* fsync() on all Append() calls
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.60@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
I20260811 11:20:35.750710 21701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3: No bootstrap required, opened a new log
I20260811 11:20:35.752518 21701 raft_consensus.cc:348] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } }
I20260811 11:20:35.752679 21701 raft_consensus.cc:729] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e840edae744743868867b853c93e7ae3, State: Initialized, Role: FOLLOWER
I20260811 11:20:35.752748 21701 consensus_queue.cc:260] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } }
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.25.61@KRBTEST.COM
I20260811 11:20:35.753022 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } } }
I20260811 11:20:35.753111 21701 sys_catalog.cc:564] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:35.753113 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:35.754792 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.745511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52468)
0811 11:20:35.745691 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.745701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:35.745735 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:35.745916 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:35.745917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:35.746067 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:35.746105 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.746107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.747090 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.747091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.747898 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.747900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.748349 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.748350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.748399 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.748836 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:35.748840 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:35.750554 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:35.753414 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.753415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.753416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.753668 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.753967 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.753968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.753969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.754036 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.754414 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:35.754415 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:35.754460 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 11:20:35.754545 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 11:20:35.754654 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260811 11:20:35.755276 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.745541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.28.25.60:45981)
0811 11:20:35.745748 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.745750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.745757 (+ 7us) server_negotiation.cc:407] Connection header received
0811 11:20:35.745780 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.745781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.745793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.745861 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.746186 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.747009 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.747977 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.748266 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.748352 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.750635 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.750636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.750637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.750645 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.753314 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.753803 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.753804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.753805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.753884 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.754102 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.754103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.754103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.754311 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.754389 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.755045 (+ 656us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.755157 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":119,"threads_started":1}
W20260811 11:20:35.755923 21885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:35.755982 21885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:35.757035 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf: Bootstrap starting.
I20260811 11:20:35.757629 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.747076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.28.25.61:35735)
0811 11:20:35.747112 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.747114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.747125 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:20:35.747155 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.747155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.747166 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.747245 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.747837 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.748501 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.749742 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.750073 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.750150 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.754011 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.754013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.754013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.754021 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.756156 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.756595 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.756596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.756597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.756660 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.756910 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.756911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.756911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.757053 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.757197 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.757447 (+ 250us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.757540 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260811 11:20:35.757858 21867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:35.758174 21867 log.cc:824] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:35.756197 21624 master_runner.cc:186] Error getting master registration for 127.28.25.61:35735: OK, instance_id { permanent_uuid: "ff39084911744df68596a14c4682198b" instance_seqno: 1786447235547869 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:20:35.758811 21624 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:20:35.759104 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf: No bootstrap required, opened a new log
I20260811 11:20:35.760466 28772 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:20:35.761236 21867 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } }
I20260811 11:20:35.761418 21867 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbeb147a460445eb8f54de49a9bfcecf, State: Initialized, Role: FOLLOWER
I20260811 11:20:35.761479 21671 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e840edae744743868867b853c93e7ae3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60712
I20260811 11:20:35.761523 21867 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } }
I20260811 11:20:35.761624 21671 raft_consensus.cc:482] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 11:20:35.761682 21671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:35.761879 21867 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:35.761827 21888 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } } }
I20260811 11:20:35.762032 21888 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:35.762609 21671 raft_consensus.cc:504] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } }
I20260811 11:20:35.762866 21671 leader_election.cc:288] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [CANDIDATE]: Term 1 election: Requested vote from peers ff39084911744df68596a14c4682198b (127.28.25.61:35735), cbeb147a460445eb8f54de49a9bfcecf (127.28.25.60:45981)
I20260811 11:20:35.763265 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e840edae744743868867b853c93e7ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cbeb147a460445eb8f54de49a9bfcecf"
I20260811 11:20:35.763402 21834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:35.764707 21834 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e840edae744743868867b853c93e7ae3 in term 1.
I20260811 11:20:35.765012 21643 leader_election.cc:302] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbeb147a460445eb8f54de49a9bfcecf, e840edae744743868867b853c93e7ae3; no voters:
I20260811 11:20:35.765232 21874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:20:35.765419 21874 raft_consensus.cc:686] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [term 1 LEADER]: Becoming Leader. State: Replica: e840edae744743868867b853c93e7ae3, State: Running, Role: LEADER
I20260811 11:20:35.765532 21874 consensus_queue.cc:237] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } }
I20260811 11:20:35.766038 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e840edae744743868867b853c93e7ae3. Latest consensus state: current_term: 1 leader_uuid: "e840edae744743868867b853c93e7ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } } }
I20260811 11:20:35.766155 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [sys.catalog]: This master's current role is: LEADER
W20260811 11:20:35.766552 21903 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:35.766656 21903 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:35.767100 21914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:20:35.767396 21914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:20:35.767976 21791 master_runner.cc:186] Error getting master registration for 127.28.25.61:35735: OK, instance_id { permanent_uuid: "ff39084911744df68596a14c4682198b" instance_seqno: 1786447235547869 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:20:35.770773 21791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:20:35.776019 21834 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [term 1 FOLLOWER]: Refusing update from remote peer e840edae744743868867b853c93e7ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:20:35.776438 21911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:20:35.778335 21888 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e840edae744743868867b853c93e7ae3. Latest consensus state: current_term: 1 leader_uuid: "e840edae744743868867b853c93e7ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } } }
I20260811 11:20:35.778505 21888 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:35.778609 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.764412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.28.25.61:35735)
0811 11:20:35.764456 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.764458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.766129 (+ 1671us) server_negotiation.cc:407] Connection header received
0811 11:20:35.766309 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.766311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.766337 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.766371 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.769614 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.770379 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.771502 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.772320 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.772392 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.773581 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.773583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.773584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.773592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.775399 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.776366 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.776368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.776368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.776439 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.776675 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.776676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.776676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.776821 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.776890 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.778406 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.778513 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 11:20:35.780022 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e840edae744743868867b853c93e7ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } } }
I20260811 11:20:35.780133 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:20:35.780357 21888 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e840edae744743868867b853c93e7ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } } }
I20260811 11:20:35.780452 21888 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:35.780565 21917 mvcc.cc:205] Tried to move back new op lower bound from 7317287877735682048 to 7317287877696565248. Current Snapshot: MvccSnapshot[applied={T|T < 7317287877735682048}]
I20260811 11:20:35.781235 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.764514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.28.25.62:45251)
0811 11:20:35.764566 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.764568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.766075 (+ 1507us) server_negotiation.cc:407] Connection header received
0811 11:20:35.766441 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.766443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.766455 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.766484 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.768117 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.768856 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.769564 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.769842 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.769907 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.771802 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.771804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.771804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.771813 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.774852 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.780072 (+ 5220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.780074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.780074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.780159 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.780419 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.780421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.780421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.780594 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.780721 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.781069 (+ 348us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.781173 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 11:20:35.786134 21784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } attempt: 3
I20260811 11:20:35.788750 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.765033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.28.25.60:45981)
0811 11:20:35.765199 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.765202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.772772 (+ 7570us) server_negotiation.cc:407] Connection header received
0811 11:20:35.773602 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.773603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.773619 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.773654 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.774352 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.775197 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.780423 (+ 5226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.780759 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.780807 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.782905 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.782907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.782907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.782915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.787262 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.787731 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.787733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.787733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.787813 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.788061 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.788062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.788062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.788194 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.788300 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.788550 (+ 250us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.788648 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
Aug 11 11:20:35 dist-test-slave-2w3w krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786447235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.25.61@KRBTEST.COM for kudu/127.28.25.60@KRBTEST.COM
I20260811 11:20:35.791523 21914 catalog_manager.cc:1347] Generated new cluster ID: 1d136ca902a144b5b48b91c397c3865c
I20260811 11:20:35.791596 21914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:20:35.794821 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.786629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.28.25.60:45981)
0811 11:20:35.786713 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.786715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.786724 (+ 9us) server_negotiation.cc:407] Connection header received
0811 11:20:35.786748 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.786749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.786760 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.786825 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.787191 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.787954 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.789004 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.789305 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.789414 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.791734 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:20:35.791735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:20:35.791735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:20:35.791743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:20:35.793505 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.793924 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.793925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.793926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.793991 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:20:35.794229 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:20:35.794230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:20:35.794230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:20:35.794393 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:20:35.794457 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:35.794624 (+ 167us) server_negotiation.cc:299] Negotiation successful
0811 11:20:35.794706 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:20:35.794834 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:35.786425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.25.60:45981 (local address 127.0.0.1:52498)
0811 11:20:35.786472 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0811 11:20:35.786501 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 11:20:35.786545 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:20:35.786839 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:20:35.786841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:20:35.787050 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:20:35.787111 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.787113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.788055 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.788057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.788874 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:20:35.788879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:20:35.789429 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:20:35.789430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:20:35.789500 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:35.790033 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:20:35.790039 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:20:35.791628 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:20:35.793600 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.793601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.793601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.793837 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.794089 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:20:35.794091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:20:35.794091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:20:35.794162 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:20:35.794467 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:20:35.794468 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:20:35.794522 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 11:20:35.794597 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 11:20:35.794724 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260811 11:20:35.796782 21784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b: Bootstrap starting.
I20260811 11:20:35.797571 21784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b: Neither blocks nor log segments found. Creating new log.
I20260811 11:20:35.797866 21784 log.cc:824] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:20:35.798991 21784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b: No bootstrap required, opened a new log
I20260811 11:20:35.799279 21914 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:20:35.800015 21914 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:20:35.801191 21784 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } }
I20260811 11:20:35.801329 21784 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff39084911744df68596a14c4682198b, State: Initialized, Role: FOLLOWER
I20260811 11:20:35.801410 21784 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } }
I20260811 11:20:35.801788 21784 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:20:35.801821 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e840edae744743868867b853c93e7ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ff39084911744df68596a14c4682198b"
I20260811 11:20:35.801784 21923 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } } }
I20260811 11:20:35.801967 21756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:20:35.803021 21756 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e840edae744743868867b853c93e7ae3 in term 1.
I20260811 11:20:35.803541 21755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [term 1 FOLLOWER]: Refusing update from remote peer e840edae744743868867b853c93e7ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260811 11:20:35.804363 21935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:20:35.804427 21935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:20:35.801962 21923 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:35.811409 21920 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260811 11:20:35.814469 21923 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e840edae744743868867b853c93e7ae3. Latest consensus state: current_term: 1 leader_uuid: "e840edae744743868867b853c93e7ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } } }
I20260811 11:20:35.814604 21923 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:35.814642 21923 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e840edae744743868867b853c93e7ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } } }
I20260811 11:20:35.814688 21923 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:20:35.815300 21938 mvcc.cc:205] Tried to move back new op lower bound from 7317287877828374528 to 7317287877696565248. Current Snapshot: MvccSnapshot[applied={T|T < 7317287877735682048}]
I20260811 11:20:35.816221 21914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3: Generated new TSK 0
I20260811 11:20:35.816453 21914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:20:35.827513 21911 sys_catalog.cc:455] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e840edae744743868867b853c93e7ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840edae744743868867b853c93e7ae3" member_type: VOTER last_known_addr { host: "127.28.25.62" port: 45251 } } peers { permanent_uuid: "ff39084911744df68596a14c4682198b" member_type: VOTER last_known_addr { host: "127.28.25.61" port: 35735 } } peers { permanent_uuid: "cbeb147a460445eb8f54de49a9bfcecf" member_type: VOTER last_known_addr { host: "127.28.25.60" port: 45981 } } }
I20260811 11:20:35.827654 21911 sys_catalog.cc:458] T 00000000000000000000000000000000 P e840edae744743868867b853c93e7ae3 [sys.catalog]: This master's current role is: LEADER
W20260811 11:20:35.846854 21706 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.845284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.28.25.62:45251)
0811 11:20:35.845336 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.845339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.846097 (+ 758us) server_negotiation.cc:407] Connection header received
0811 11:20:35.846209 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.846211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.846224 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.846256 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.846757 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60730: BlockingRecv error: recv got EOF from 127.0.0.1:60730 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260811 11:20:35.847720 21871 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.845776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.28.25.60:45981)
0811 11:20:35.845820 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.845822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.847149 (+ 1327us) server_negotiation.cc:407] Connection header received
0811 11:20:35.847254 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.847256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.847267 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.847298 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.847653 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52500: BlockingRecv error: recv got EOF from 127.0.0.1:52500 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260811 11:20:35.848908 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.848192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.28.25.61:35735)
0811 11:20:35.848239 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.848241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.848364 (+ 123us) server_negotiation.cc:407] Connection header received
0811 11:20:35.848459 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.848460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.848471 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.848500 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.848831 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40594: BlockingRecv error: recv got EOF from 127.0.0.1:40594 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260811 11:20:35.851519 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.850627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40596 (local address 127.28.25.61:35735)
0811 11:20:35.850672 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.850674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.850874 (+ 200us) server_negotiation.cc:407] Connection header received
0811 11:20:35.851047 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.851048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.851072 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.851104 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.851463 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40596: BlockingRecv error: recv got EOF from 127.0.0.1:40596 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260811 11:20:35.854899 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.854022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.28.25.61:35735)
0811 11:20:35.854061 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.854063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.854299 (+ 236us) server_negotiation.cc:407] Connection header received
0811 11:20:35.854401 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.854402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.854416 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.854448 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.854830 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40608: BlockingRecv error: recv got EOF from 127.0.0.1:40608 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260811 11:20:35.859287 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.858306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.28.25.61:35735)
0811 11:20:35.858351 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.858353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.858597 (+ 244us) server_negotiation.cc:407] Connection header received
0811 11:20:35.858703 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.858704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.858716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.858747 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.859226 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40620: BlockingRecv error: recv got EOF from 127.0.0.1:40620 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260811 11:20:35.865581 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.864673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.28.25.61:35735)
0811 11:20:35.864717 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.864719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.864983 (+ 264us) server_negotiation.cc:407] Connection header received
0811 11:20:35.865090 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.865091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.865103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.865134 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.865520 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40626: BlockingRecv error: recv got EOF from 127.0.0.1:40626 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260811 11:20:35.876225 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.875286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.28.25.61:35735)
0811 11:20:35.875333 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.875335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.875594 (+ 259us) server_negotiation.cc:407] Connection header received
0811 11:20:35.875702 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.875703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.875715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.875745 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.876149 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40630: BlockingRecv error: recv got EOF from 127.0.0.1:40630 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260811 11:20:35.894783 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.893737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.28.25.61:35735)
0811 11:20:35.893783 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.893785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.894081 (+ 296us) server_negotiation.cc:407] Connection header received
0811 11:20:35.894197 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.894199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.894211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.894243 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.894700 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40646: BlockingRecv error: recv got EOF from 127.0.0.1:40646 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260811 11:20:35.929454 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.928380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.28.25.61:35735)
0811 11:20:35.928432 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.928435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.928716 (+ 281us) server_negotiation.cc:407] Connection header received
0811 11:20:35.928823 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.928825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.928838 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.928936 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.929386 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40648: BlockingRecv error: recv got EOF from 127.0.0.1:40648 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260811 11:20:35.996026 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:35.995116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.28.25.61:35735)
0811 11:20:35.995195 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:35.995198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:35.995371 (+ 173us) server_negotiation.cc:407] Connection header received
0811 11:20:35.995457 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:35.995459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:35.995473 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:35.995548 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:35.995936 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40654: BlockingRecv error: recv got EOF from 127.0.0.1:40654 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260811 11:20:36.126462 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:36.125588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40666 (local address 127.28.25.61:35735)
0811 11:20:36.125694 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:36.125696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:36.125838 (+ 142us) server_negotiation.cc:407] Connection header received
0811 11:20:36.125872 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:36.125873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:36.125884 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:36.125948 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:36.126370 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40666: BlockingRecv error: recv got EOF from 127.0.0.1:40666 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260811 11:20:36.385104 21788 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:20:36.384333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.28.25.61:35735)
0811 11:20:36.384416 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:36.384419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:36.384430 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:20:36.384507 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:36.384508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:36.384519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:36.384563 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:20:36.385018 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40676: BlockingRecv error: recv got EOF from 127.0.0.1:40676 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 11:20:36.767184 21903 catalog_manager.cc:1259] Loaded cluster ID: 1d136ca902a144b5b48b91c397c3865c
I20260811 11:20:36.767268 21903 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf: loading cluster ID for follower catalog manager: success
I20260811 11:20:36.768152 21903 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf: acquiring CA information for follower catalog manager: success
I20260811 11:20:36.768508 21903 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cbeb147a460445eb8f54de49a9bfcecf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:20:36.804867 21935 catalog_manager.cc:1259] Loaded cluster ID: 1d136ca902a144b5b48b91c397c3865c
I20260811 11:20:36.804963 21935 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b: loading cluster ID for follower catalog manager: success
I20260811 11:20:36.805763 21935 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b: acquiring CA information for follower catalog manager: success
I20260811 11:20:36.806041 21935 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff39084911744df68596a14c4682198b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:20:36.901377 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:36.898447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.28.25.61:35735)
0811 11:20:36.898593 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:36.898595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:36.898681 (+ 86us) server_negotiation.cc:407] Connection header received
0811 11:20:36.898785 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:36.898786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:36.898868 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:36.898967 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:36.899153 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:36.899673 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:36.900510 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:36.900719 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:36.900796 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:36.900973 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:36.901085 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:36.901219 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:36.901236 (+ 17us) server_negotiation.cc:299] Negotiation successful
0811 11:20:36.901260 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260811 11:20:36.906117 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:36.903482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.28.25.60:45981)
0811 11:20:36.903603 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:36.903605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:36.903650 (+ 45us) server_negotiation.cc:407] Connection header received
0811 11:20:36.903724 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:36.903725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:36.903764 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:36.903838 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:36.904029 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:36.904537 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:36.905319 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:36.905522 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:36.905577 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:36.905758 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:36.905878 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:36.905937 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:36.906028 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 11:20:36.906047 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260811 11:20:36.910975 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:36.908042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.28.25.61:35735)
0811 11:20:36.908097 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:36.908099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:36.908350 (+ 251us) server_negotiation.cc:407] Connection header received
0811 11:20:36.908462 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:36.908463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:36.908475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:36.908530 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:36.908672 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:36.909166 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:36.910122 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:36.910430 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:36.910504 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:36.910661 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:36.910697 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:36.910723 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:36.910835 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 11:20:36.910857 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 11:20:36.911144 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:36.908047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.28.25.60:45981)
0811 11:20:36.908129 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:36.908131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:36.908341 (+ 210us) server_negotiation.cc:407] Connection header received
0811 11:20:36.908404 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:36.908405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:36.908417 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:36.908468 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:36.908636 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:36.909375 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:36.910326 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:36.910609 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:36.910673 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:36.910835 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:36.910867 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:36.910918 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:36.911028 (+ 110us) server_negotiation.cc:299] Negotiation successful
0811 11:20:36.911052 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 11:20:36.911713 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:20:36.908057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.28.25.62:45251)
0811 11:20:36.908262 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 11:20:36.908265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:20:36.908434 (+ 169us) server_negotiation.cc:407] Connection header received
0811 11:20:36.908484 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:20:36.908485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:20:36.908538 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:20:36.908623 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:20:36.908732 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:36.909475 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:36.910581 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:20:36.910908 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:20:36.911001 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:20:36.911191 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:20:36.911324 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:20:36.911394 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 11:20:36.911506 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 11:20:36.911623 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":61,"threads_started":1}
I20260811 11:20:36.913519 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 21624
I20260811 11:20:36.921752 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 21710
I20260811 11:20:36.930377 28772 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR10UZg/build/release/bin/kudu with pid 21791
2026-08-11T11:20:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2936 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5975 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (324932 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS